2024-09-16T03:56:47.789 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T03:56:47.813 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:56:47.901 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966 branch: squid description: rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905966' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - mon.a - - osd.4 - osd.5 - osd.6 - osd.7 - mon.b - mgr.x - - osd.8 - osd.9 - osd.10 - osd.11 - mon.c seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNVGk+mTN4O/+B5mgT/ywEsv0T8BJL55lyldEerkLi1YKLJUTMZ/DoelrQRNVFsliLqbu7djZNp/ZU2PzVVTU0o= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPldhjKpiowGAQ3cIOs6Lcq2K5LV+nejpkxm7Dz2SS/Dalpd6p7+7typ5Eajd7xIV33sgRL3a4IwJQfS+1PCgjA= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvw05pPvP+wC8ydK4DsWixLJPlEJTGPSP9ZIXCNc1Josv6Kd4FwyCc2T1tQlQbBEGpwQDdNzwhINTGU+vyKrIs= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: crush-failure-domain: osd k: 4 l: 3 m: 2 name: lrcprofile plugin: lrc objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T03:56:47.902 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T03:56:47.902 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T03:56:47.903 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T03:56:47.905 INFO:teuthology.task.internal:Checking packages... 2024-09-16T03:56:47.928 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T03:56:47.928 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T03:56:47.928 INFO:teuthology.packaging:ref: None 2024-09-16T03:56:47.928 INFO:teuthology.packaging:tag: None 2024-09-16T03:56:47.928 INFO:teuthology.packaging:branch: squid 2024-09-16T03:56:47.928 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:56:47.928 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T03:56:48.158 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T03:56:48.160 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T03:56:48.169 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T03:56:48.169 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T03:56:48.183 INFO:teuthology.task.internal:Saving configuration 2024-09-16T03:56:48.202 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T03:56:48.213 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T03:56:48.239 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:51:53.865397', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNVGk+mTN4O/+B5mgT/ywEsv0T8BJL55lyldEerkLi1YKLJUTMZ/DoelrQRNVFsliLqbu7djZNp/ZU2PzVVTU0o='} 2024-09-16T03:56:48.263 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:51:53.867238', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPldhjKpiowGAQ3cIOs6Lcq2K5LV+nejpkxm7Dz2SS/Dalpd6p7+7typ5Eajd7xIV33sgRL3a4IwJQfS+1PCgjA='} 2024-09-16T03:56:48.288 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:51:53.868788', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvw05pPvP+wC8ydK4DsWixLJPlEJTGPSP9ZIXCNc1Josv6Kd4FwyCc2T1tQlQbBEGpwQDdNzwhINTGU+vyKrIs='} 2024-09-16T03:56:48.288 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T03:56:48.297 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2024-09-16T03:56:48.297 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2024-09-16T03:56:48.297 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2024-09-16T03:56:48.297 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T03:56:48.461 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa853fd7ac0>, signals=[15]) 2024-09-16T03:56:48.461 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T03:56:48.469 INFO:teuthology.task.internal:Opening connections... 2024-09-16T03:56:48.470 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T03:56:48.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:56:48.554 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T03:56:48.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:56:48.645 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T03:56:48.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:56:48.739 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T03:56:48.746 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-09-16T03:56:48.766 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-09-16T03:56:48.767 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:56:48.825 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:56:48.826 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:56:48.826 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:56:48.826 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:56:48.826 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-09-16T03:56:48.849 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-09-16T03:56:48.868 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-09-16T03:56:48.868 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-09-16T03:56:48.927 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-09-16T03:56:48.927 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-09-16T03:56:48.927 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-09-16T03:56:48.927 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:56:48.927 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-09-16T03:56:48.927 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:56:48.927 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:56:48.927 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-09-16T03:56:48.928 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:56:48.928 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:56:48.928 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:56:48.928 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:56:48.928 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:56:48.928 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:56:48.928 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-09-16T03:56:48.950 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-09-16T03:56:48.968 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-09-16T03:56:48.969 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-09-16T03:56:49.027 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-09-16T03:56:49.027 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-09-16T03:56:49.027 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-09-16T03:56:49.027 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:56:49.027 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-09-16T03:56:49.027 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:56:49.028 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:56:49.028 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-09-16T03:56:49.028 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:56:49.028 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:56:49.028 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:56:49.028 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:56:49.028 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:56:49.028 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:56:49.028 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-09-16T03:56:49.049 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T03:56:49.060 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T03:56:49.067 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T03:56:49.067 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:56:49.070 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:56:49.072 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:56:49.089 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T03:56:49.097 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T03:56:49.097 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:56:49.127 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:56:49.130 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:56:49.158 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T03:56:49.230 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T03:56:49.250 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T03:56:49.250 INFO:teuthology.task.kernel:config {'osd.0': {'kdb': 1, 'sha1': 'distro'}, 'osd.4': {'kdb': 1, 'sha1': 'distro'}, 'osd.8': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T03:56:49.251 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:56:49.252 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:56:49.252 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:56:49.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:56:49.270 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-16T03:56:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:56:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:56:49.272 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-09-16T03:56:49.272 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-16T03:56:49.327 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:56:49.327 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.14.0-503.el9.x86_64 2024-09-16T03:56:49.327 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum install -y kernel 2024-09-16T03:56:49.330 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:56:49.330 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-503.el9.x86_64 2024-09-16T03:56:49.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-09-16T03:56:49.331 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:56:49.331 INFO:teuthology.task.kernel:Running kernel on smithi194: 5.14.0-503.el9.x86_64 2024-09-16T03:56:49.332 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-09-16T03:56:51.742 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-16T03:56:51.980 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-16T03:56:52.901 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-16T03:56:53.775 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-16T03:56:53.801 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-16T03:56:54.952 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-16T03:56:58.257 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-16T03:56:58.377 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.5 MB 00:00 2024-09-16T03:56:59.799 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 19 kB 00:00 2024-09-16T03:56:59.835 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 82 kB/s | 19 kB 00:00 2024-09-16T03:57:00.428 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-16T03:57:00.906 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-16T03:57:00.945 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-16T03:57:02.678 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 19 kB 00:01 2024-09-16T03:57:03.691 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-16T03:57:06.270 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T03:57:06.535 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-16T03:57:08.293 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:57:08.293 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:57:08.392 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T03:57:08.399 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T03:57:08.399 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-09-16T03:57:08.400 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T03:57:08.401 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 79 M 2024-09-16T03:57:08.401 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 126 M 2024-09-16T03:57:08.401 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T03:57:08.643 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:57:08.643 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:57:08.738 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T03:57:08.748 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T03:57:08.749 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 79 M 2024-09-16T03:57:08.750 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 126 M 2024-09-16T03:57:08.750 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T03:57:08.796 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 77 kB/s | 23 kB 00:00 2024-09-16T03:57:09.086 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 270 kB/s | 23 kB 00:00 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T03:57:09.461 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-16T03:57:09.878 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 39 MB/s | 35 MB 00:00 2024-09-16T03:57:10.237 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 25 MB/s | 29 MB 00:01 2024-09-16T03:57:10.238 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:57:10.238 INFO:teuthology.orchestra.run.smithi016.stdout:Total 53 MB/s | 79 MB 00:01 2024-09-16T03:57:10.953 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T03:57:11.211 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:57:11.211 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:57:11.309 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T03:57:11.317 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T03:57:11.317 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-09-16T03:57:11.317 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T03:57:11.317 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T03:57:11.317 INFO:teuthology.orchestra.run.smithi174.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-16T03:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T03:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T03:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T03:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout:Install 4 Packages 2024-09-16T03:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T03:57:11.319 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 79 M 2024-09-16T03:57:11.319 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 126 M 2024-09-16T03:57:11.319 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T03:57:11.343 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T03:57:11.344 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T03:57:11.931 INFO:teuthology.orchestra.run.smithi174.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 39 kB/s | 23 kB 00:00 2024-09-16T03:57:12.303 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T03:57:12.304 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T03:57:12.881 INFO:teuthology.orchestra.run.smithi174.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-16T03:57:13.390 INFO:teuthology.orchestra.run.smithi174.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-16T03:57:14.248 INFO:teuthology.orchestra.run.smithi174.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-16T03:57:14.248 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:57:14.249 INFO:teuthology.orchestra.run.smithi174.stdout:Total 27 MB/s | 79 MB 00:02 2024-09-16T03:57:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T03:57:14.929 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T03:57:15.256 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T03:57:15.256 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T03:57:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:57:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:57:16.224 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T03:57:16.225 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T03:57:18.432 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:57:18.558 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:57:18.689 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T03:57:19.358 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:57:19.433 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:57:22.221 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:57:22.350 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:57:23.674 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:57:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:57:23.795 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 1.0 MB/s | 16 MB 00:15 2024-09-16T03:57:27.174 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:57:27.225 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:57:28.856 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:57:32.066 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:57:43.259 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 848 kB/s | 29 MB 00:34 2024-09-16T03:57:50.475 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 843 kB/s | 35 MB 00:41 2024-09-16T03:57:50.475 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:57:50.475 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.9 MB/s | 79 MB 00:42 2024-09-16T03:57:51.156 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T03:57:51.525 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T03:57:51.525 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T03:57:52.462 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T03:57:52.462 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T03:57:54.893 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T03:57:55.534 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:57:55.627 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:57:58.420 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:57:58.545 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:58:03.399 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:58:03.510 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:07.857 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:07.898 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:58:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:58:08.329 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:58:08.365 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:09.166 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:09.167 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T03:58:09.167 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T03:58:09.167 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:58:09.167 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:58:09.167 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:58:09.167 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:58:09.167 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T03:58:09.167 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T03:58:09.543 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:58:09.634 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:58:09.634 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:58:09.635 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:58:09.635 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:58:09.635 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:58:09.635 INFO:teuthology.task.kernel:Installing distro kernel on osd.0... 2024-09-16T03:58:09.635 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:58:09.635 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:58:09.694 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:58:09.695 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:58:09.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-09-16T03:58:09.736 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-09-16T03:58:09.736 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-09-16T03:58:09.736 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-09-16T03:58:09.736 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 82.el9 2024-09-16T03:58:09.736 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8274670 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-09-16T03:58:09.737 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:58:09.738 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-09-16T03:58:09.738 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-09-16T03:58:09.738 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T03:58:09.738 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:58:09.738 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:58:09.738 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:58:09.738 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-09-16T03:58:09.738 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T03:58:09.738 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:58:09.739 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:58:09.739 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:58:09.740 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:58:10.374 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:10.416 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:10.452 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-09-16T03:58:10.746 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:10.746 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:58:10.746 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:58:10.746 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:58:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:11.584 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T03:58:11.584 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T03:58:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:58:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:58:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:58:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:58:11.585 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T03:58:11.585 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T03:58:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:58:11.755 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-09-16T03:58:11.763 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-16T03:58:11.778 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.g0aLVuVMBp 2024-09-16T03:58:11.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.g0aLVuVMBp 2024-09-16T03:58:11.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.g0aLVuVMBp 2024-09-16T03:58:11.929 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:58:11.994 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.g0aLVuVMBp is 6KB 2024-09-16T03:58:12.013 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:58:12.014 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:58:12.014 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:58:12.014 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:58:12.014 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:58:12.015 INFO:teuthology.task.kernel:Installing distro kernel on osd.4... 2024-09-16T03:58:12.015 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:58:12.015 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:58:12.016 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.g0aLVuVMBp 2024-09-16T03:58:12.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:58:12.074 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:58:12.074 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:58:12.074 DEBUG:teuthology.orchestra.run.smithi174:> sudo rpm -qi grub2-tools 2024-09-16T03:58:12.093 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:58:12.094 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:58:12.094 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:58:12.094 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:58:12.095 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:58:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Name : grub2-tools 2024-09-16T03:58:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Epoch : 1 2024-09-16T03:58:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Version : 2.06 2024-09-16T03:58:12.106 INFO:teuthology.orchestra.run.smithi174.stdout:Release : 82.el9 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: x86_64 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Group : Unspecified 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Size : 8274670 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:License : GPLv3+ 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Packager : builder@centos.org 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Vendor : CentOS 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Summary : Support tools for GRUB. 2024-09-16T03:58:12.107 INFO:teuthology.orchestra.run.smithi174.stdout:Description : 2024-09-16T03:58:12.108 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T03:58:12.108 INFO:teuthology.orchestra.run.smithi174.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:58:12.108 INFO:teuthology.orchestra.run.smithi174.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:58:12.108 INFO:teuthology.orchestra.run.smithi174.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:58:12.108 INFO:teuthology.orchestra.run.smithi174.stdout:hardware devices. 2024-09-16T03:58:12.108 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T03:58:12.108 INFO:teuthology.orchestra.run.smithi174.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:58:12.109 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:58:12.109 INFO:teuthology.task.kernel:Updating grub on smithi174 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:58:12.109 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:58:12.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-09-16T03:58:12.828 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-09-16T03:58:14.167 INFO:teuthology.orchestra.run.smithi174.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:58:14.183 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-09-16T03:58:14.185 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-09-16T03:58:14.200 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.psnwWKl04i 2024-09-16T03:58:14.201 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.psnwWKl04i 2024-09-16T03:58:14.267 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.psnwWKl04i 2024-09-16T03:58:14.414 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.psnwWKl04i is 6KB 2024-09-16T03:58:14.434 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.psnwWKl04i 2024-09-16T03:58:14.449 DEBUG:teuthology.orchestra.run.smithi174:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi174.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi174.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi174.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi174.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:58:14.520 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:58:14.665 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-09-16T03:58:14.739 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-09-16T03:58:42.230 INFO:teuthology.task.kernel:Checking client osd.0 for new kernel version... 2024-09-16T03:58:42.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-16T03:58:42.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:58:44.668 INFO:teuthology.task.kernel:Checking client osd.4 for new kernel version... 2024-09-16T03:58:44.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-16T03:58:44.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:58:46.602 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:46.641 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:47.009 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:47.009 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:58:47.009 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:58:47.009 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:58:47.883 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:58:47.884 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T03:58:47.884 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T03:58:47.884 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:58:47.884 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:58:47.884 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:58:47.884 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:58:47.884 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T03:58:47.884 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T03:58:48.216 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:58:48.298 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:58:48.298 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:58:48.298 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:58:48.298 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:58:48.298 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:58:48.299 INFO:teuthology.task.kernel:Installing distro kernel on osd.8... 2024-09-16T03:58:48.299 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:58:48.299 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:58:48.357 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:58:48.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:58:48.357 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-09-16T03:58:48.390 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-09-16T03:58:48.390 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-09-16T03:58:48.390 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.06 2024-09-16T03:58:48.390 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 82.el9 2024-09-16T03:58:48.390 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:Group : Unspecified 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 8274670 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : builder@centos.org 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-09-16T03:58:48.391 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:58:48.392 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-09-16T03:58:48.392 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-09-16T03:58:48.392 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T03:58:48.392 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:58:48.392 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:58:48.392 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:58:48.392 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-09-16T03:58:48.392 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T03:58:48.393 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:58:48.394 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:58:48.394 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:58:48.394 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:58:49.089 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-09-16T03:58:50.450 INFO:teuthology.orchestra.run.smithi194.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:58:50.488 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-09-16T03:58:50.490 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.kywUnxDims 2024-09-16T03:58:50.511 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kywUnxDims 2024-09-16T03:58:50.581 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.kywUnxDims 2024-09-16T03:58:50.732 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.kywUnxDims is 6KB 2024-09-16T03:58:50.758 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.kywUnxDims 2024-09-16T03:58:50.773 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:58:50.836 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:58:50.836 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:58:50.836 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:58:50.836 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:58:50.838 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:58:51.001 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-09-16T03:59:21.033 INFO:teuthology.task.kernel:Checking client osd.8 for new kernel version... 2024-09-16T03:59:21.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-16T03:59:21.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:59:42.253 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:59:44.673 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:59:51.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-16T03:59:51.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:59:51.759 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-16T03:59:52.113 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-16T03:59:52.113 INFO:teuthology.task.kernel:Checking kernel version of osd.0, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:59:52.113 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-16T03:59:52.130 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:59:52.130 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:59:52.130 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:59:52.130 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:59:53.131 INFO:teuthology.task.kernel:Enabling kdb on osd.0... 2024-09-16T03:59:53.131 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:59:53.249 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-09-16T03:59:53.288 DEBUG:teuthology.parallel:result is None 2024-09-16T03:59:53.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-16T03:59:53.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:59:54.075 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-16T03:59:54.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-16T03:59:54.541 INFO:teuthology.task.kernel:Checking kernel version of osd.4, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:59:54.542 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-16T03:59:54.556 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:59:54.556 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:59:54.556 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:59:54.557 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:59:55.557 INFO:teuthology.task.kernel:Enabling kdb on osd.4... 2024-09-16T03:59:55.558 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:59:55.635 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-09-16T03:59:55.660 DEBUG:teuthology.parallel:result is None 2024-09-16T04:00:21.038 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T04:00:30.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-16T04:00:30.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:00:30.338 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-09-16T04:00:30.755 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-16T04:00:30.755 INFO:teuthology.task.kernel:Checking kernel version of osd.8, want "5.14.0-505.el9.x86_64"... 2024-09-16T04:00:30.756 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-09-16T04:00:30.770 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-505.el9.x86_64 2024-09-16T04:00:30.770 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T04:00:30.770 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T04:00:30.771 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T04:00:31.772 INFO:teuthology.task.kernel:Enabling kdb on osd.8... 2024-09-16T04:00:31.772 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T04:00:31.851 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-09-16T04:00:31.870 DEBUG:teuthology.parallel:result is None 2024-09-16T04:00:31.871 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T04:00:31.880 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T04:00:31.880 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:00:31.884 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:00:31.887 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T04:00:31.973 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T04:00:31.980 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T04:00:31.989 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T04:00:31.990 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:00:31.992 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:00:31.995 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T04:00:32.066 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T04:00:32.073 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T04:00:32.073 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:00:32.076 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:00:32.078 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T04:00:32.106 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:00:32.109 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:00:32.117 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:00:32.120 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:00:32.126 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:00:32.138 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T04:00:32.139 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T04:00:32.147 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T04:00:32.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:00:32.163 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:00:32.165 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T04:00:32.191 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T04:00:32.200 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T04:00:32.201 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:00:32.234 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:00:32.236 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T04:00:32.252 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:00:32.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:00:32.394 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:00:32.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:00:32.481 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:00:32.481 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:00:32.546 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:00:32.589 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:00:32.667 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:00:32.705 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:00:32.752 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:00:32.753 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:00:32.819 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:00:32.881 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T04:00:32.979 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:00:33.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T04:00:33.144 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:00:33.144 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T04:00:33.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-09-16T04:00:33.219 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-09-16T04:00:33.222 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-09-16T04:00:33.276 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:00:33.280 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:00:33.361 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:00:33.668 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T04:00:33.677 INFO:teuthology.task.internal:Starting timer... 2024-09-16T04:00:33.677 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T04:00:33.694 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T04:00:33.709 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T04:00:33.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-09-16T04:00:33.779 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-09-16T04:00:33.781 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-09-16T04:00:33.850 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-09-16T04:00:33.853 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-09-16T04:00:33.976 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-09-16T04:00:33.978 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T04:00:33.979 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-09-16T04:00:34.007 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-09-16T04:00:34.007 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-09-16T04:00:34.049 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-09-16T04:00:34.049 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-09-16T04:00:34.131 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-09-16T04:00:34.131 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-09-16T04:00:34.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:00:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:34.161 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:00:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:34.192 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:00:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:00:34.218 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T04:00:34.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-09-16T04:00:34.265 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-09-16T04:00:34.299 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-09-16T04:00:34.349 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T04:00:34.364 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T04:00:34.367 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T04:00:34.519 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T04:00:34.532 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T04:00:34.533 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi174.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T04:04:20.293 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-09-16T04:04:20.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-16T04:04:20.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:04:20.372 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-16T04:04:20.453 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-16T04:04:20.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-16T04:04:20.454 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:04:20.530 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-16T04:04:20.614 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-16T04:04:20.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-16T04:04:20.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:04:20.699 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-09-16T04:04:20.741 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-16T04:04:20.741 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T04:04:20.751 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T04:04:20.751 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:04:20.752 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:04:20.754 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:04:20.755 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:04:20.757 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:04:20.757 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:04:20.786 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:04:20.787 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:04:20.802 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:04:20.803 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:04:20.812 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:04:20.826 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:04:20.830 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-09-16T04:04:20.830 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-09-16T04:04:20.841 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-09-16T04:04:20.843 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-09-16T04:04:20.851 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-09-16T04:04:20.856 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:04:20.857 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:04:20.862 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-09-16T04:04:20.871 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:04:20.872 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:04:20.876 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:04:20.890 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:04:20.927 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:04:20.931 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:04:20.931 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-09-16T04:04:20.931 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.931 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.932 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.932 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:04:20.936 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:04:20.936 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-16T04:04:20.937 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.937 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.937 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.937 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.950 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:04:20.953 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:04:20.953 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-09-16T04:04:20.953 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.953 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.953 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.953 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:04:20.954 INFO:teuthology.run_tasks:Running task install... 2024-09-16T04:04:20.964 DEBUG:teuthology.task.install:project ceph 2024-09-16T04:04:20.964 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T04:04:20.964 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T04:04:20.964 INFO:teuthology.task.install:Using flavor: default 2024-09-16T04:04:20.970 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T04:04:20.970 INFO:teuthology.task.install:extra packages: [] 2024-09-16T04:04:20.971 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:04:20.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:04:20.973 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:04:20.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:04:20.975 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:04:20.975 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:04:21.111 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T04:04:21.112 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T04:04:21.125 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T04:04:21.125 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T04:04:21.128 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T04:04:21.128 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T04:04:21.234 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:04:21.234 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:04:21.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:04:21.246 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:04:21.246 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:04:21.246 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:04:21.256 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:04:21.256 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:04:21.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:04:21.262 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:04:21.262 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:04:21.272 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:04:21.273 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:04:21.282 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:04:21.282 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:04:21.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:04:21.338 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:04:21.347 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:04:21.405 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:04:21.413 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:04:21.422 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:04:21.471 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-09-16T04:04:21.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-09-16T04:04:21.477 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-09-16T04:04:21.479 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-09-16T04:04:21.486 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-09-16T04:04:21.488 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-09-16T04:04:21.706 INFO:teuthology.orchestra.run.smithi016.stdout:45 files removed 2024-09-16T04:04:21.715 INFO:teuthology.orchestra.run.smithi174.stdout:45 files removed 2024-09-16T04:04:21.719 INFO:teuthology.orchestra.run.smithi194.stdout:45 files removed 2024-09-16T04:04:21.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw 2024-09-16T04:04:21.743 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw 2024-09-16T04:04:21.746 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-09-16T04:04:22.309 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 304 kB/s | 83 kB 00:00 2024-09-16T04:04:22.331 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 277 kB/s | 83 kB 00:00 2024-09-16T04:04:22.343 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 268 kB/s | 83 kB 00:00 2024-09-16T04:04:22.601 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-16T04:04:22.623 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-16T04:04:22.628 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T04:04:22.993 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-09-16T04:04:23.007 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-09-16T04:04:23.019 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-09-16T04:04:23.460 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-09-16T04:04:23.628 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-16T04:04:24.848 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-16T04:04:25.686 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 19 MB/s | 20 MB 00:01 2024-09-16T04:04:30.232 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 4.8 MB/s | 20 MB 00:04 2024-09-16T04:04:30.337 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.5 MB 00:00 2024-09-16T04:04:31.895 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 20 MB 00:07 2024-09-16T04:04:31.954 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-16T04:04:32.934 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-16T04:04:34.816 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-16T04:04:36.392 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 19 kB 00:00 2024-09-16T04:04:37.638 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.5 MB 00:01 2024-09-16T04:04:37.845 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-16T04:04:38.589 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-16T04:04:39.180 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 19 kB 00:00 2024-09-16T04:04:40.150 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-16T04:04:41.119 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:04:41.131 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:04:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:04:41.131 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:04:41.131 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:04:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T04:04:41.131 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-09-16T04:04:41.131 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T04:04:41.132 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T04:04:41.133 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout:Install 30 Packages 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2024-09-16T04:04:41.134 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:04:41.135 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 60 M 2024-09-16T04:04:41.135 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:04:41.883 INFO:teuthology.orchestra.run.smithi016.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 12 MB/s | 5.4 MB 00:00 2024-09-16T04:04:41.941 INFO:teuthology.orchestra.run.smithi016.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 433 kB/s | 25 kB 00:00 2024-09-16T04:04:42.033 INFO:teuthology.orchestra.run.smithi016.stdout:(3/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 8.0 MB/s | 751 kB 00:00 2024-09-16T04:04:42.117 INFO:teuthology.orchestra.run.smithi016.stdout:(4/32): libradosstriper1-19.1.1-326.g8921c8ef.e 5.9 MB/s | 506 kB 00:00 2024-09-16T04:04:42.434 INFO:teuthology.orchestra.run.smithi016.stdout:(5/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 11 MB/s | 11 MB 00:00 2024-09-16T04:04:42.667 INFO:teuthology.orchestra.run.smithi016.stdout:(6/32): ceph-common-19.1.1-326.g8921c8ef.el9.x8 17 MB/s | 21 MB 00:01 2024-09-16T04:04:42.684 INFO:teuthology.orchestra.run.smithi016.stdout:(7/32): python3-ceph-argparse-19.1.1-326.g8921c 180 kB/s | 45 kB 00:00 2024-09-16T04:04:42.759 INFO:teuthology.orchestra.run.smithi016.stdout:(8/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_64 8.3 MB/s | 5.3 MB 00:00 2024-09-16T04:04:42.785 INFO:teuthology.orchestra.run.smithi016.stdout:(9/32): python3-ceph-common-19.1.1-326.g8921c8e 1.1 MB/s | 136 kB 00:00 2024-09-16T04:04:42.810 INFO:teuthology.orchestra.run.smithi016.stdout:(10/32): python3-cephfs-19.1.1-326.g8921c8ef.el 1.3 MB/s | 161 kB 00:00 2024-09-16T04:04:42.835 INFO:teuthology.orchestra.run.smithi016.stdout:(11/32): python3-rados-19.1.1-326.g8921c8ef.el9 4.1 MB/s | 320 kB 00:00 2024-09-16T04:04:42.860 INFO:teuthology.orchestra.run.smithi016.stdout:(12/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 3.9 MB/s | 300 kB 00:00 2024-09-16T04:04:42.886 INFO:teuthology.orchestra.run.smithi016.stdout:(13/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.3 MB/s | 99 kB 00:00 2024-09-16T04:04:42.971 INFO:teuthology.orchestra.run.smithi016.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 529 kB/s | 45 kB 00:00 2024-09-16T04:04:42.997 INFO:teuthology.orchestra.run.smithi016.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 787 kB/s | 107 kB 00:00 2024-09-16T04:04:43.022 INFO:teuthology.orchestra.run.smithi016.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 994 kB/s | 160 kB 00:00 2024-09-16T04:04:43.064 INFO:teuthology.orchestra.run.smithi016.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-09-16T04:04:43.089 INFO:teuthology.orchestra.run.smithi016.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 630 kB/s | 42 kB 00:00 2024-09-16T04:04:43.115 INFO:teuthology.orchestra.run.smithi016.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-16T04:04:43.140 INFO:teuthology.orchestra.run.smithi016.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 714 kB/s | 36 kB 00:00 2024-09-16T04:04:43.174 INFO:teuthology.orchestra.run.smithi016.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-16T04:04:43.199 INFO:teuthology.orchestra.run.smithi016.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-09-16T04:04:43.258 INFO:teuthology.orchestra.run.smithi016.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-16T04:04:43.283 INFO:teuthology.orchestra.run.smithi016.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 385 kB/s | 26 kB 00:00 2024-09-16T04:04:43.308 INFO:teuthology.orchestra.run.smithi016.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-16T04:04:43.333 INFO:teuthology.orchestra.run.smithi016.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T04:04:43.375 INFO:teuthology.orchestra.run.smithi016.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T04:04:43.417 INFO:teuthology.orchestra.run.smithi016.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-16T04:04:43.468 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T04:04:43.501 INFO:teuthology.orchestra.run.smithi016.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-16T04:04:43.601 INFO:teuthology.orchestra.run.smithi016.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-16T04:04:43.660 INFO:teuthology.orchestra.run.smithi016.stdout:(31/32): librados2-19.1.1-326.g8921c8ef.el9.x86 14 MB/s | 3.4 MB 00:00 2024-09-16T04:04:43.768 INFO:teuthology.orchestra.run.smithi016.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-16T04:04:43.773 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:43.773 INFO:teuthology.orchestra.run.smithi016.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-16T04:04:43.875 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:04:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:04:43.915 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:04:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:04:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:04:45.597 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T04:04:45.852 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:04:45.865 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:04:45.865 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:04:45.865 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:04:45.865 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:04:45.865 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T04:04:45.865 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-09-16T04:04:45.865 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T04:04:45.865 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T04:04:45.866 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T04:04:45.867 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout:Install 30 Packages 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-09-16T04:04:45.868 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:04:45.869 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 60 M 2024-09-16T04:04:45.869 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:04:46.148 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:04:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T04:04:46.249 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T04:04:46.282 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T04:04:46.313 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:46.335 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:46.524 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:46.586 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:46.803 INFO:teuthology.orchestra.run.smithi194.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-16T04:04:46.878 INFO:teuthology.orchestra.run.smithi194.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 337 kB/s | 25 kB 00:00 2024-09-16T04:04:46.974 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T04:04:47.045 INFO:teuthology.orchestra.run.smithi194.stdout:(3/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 14 MB/s | 11 MB 00:00 2024-09-16T04:04:47.050 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:47.086 INFO:teuthology.orchestra.run.smithi194.stdout:(4/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 3.5 MB/s | 751 kB 00:00 2024-09-16T04:04:47.120 INFO:teuthology.orchestra.run.smithi194.stdout:(5/32): libradosstriper1-19.1.1-326.g8921c8ef.e 6.6 MB/s | 506 kB 00:00 2024-09-16T04:04:47.135 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:47.179 INFO:teuthology.orchestra.run.smithi194.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 774 kB/s | 45 kB 00:00 2024-09-16T04:04:47.226 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T04:04:47.252 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:47.270 INFO:teuthology.orchestra.run.smithi194.stdout:(7/32): python3-ceph-common-19.1.1-326.g8921c8e 1.5 MB/s | 136 kB 00:00 2024-09-16T04:04:47.307 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:47.336 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T04:04:47.346 INFO:teuthology.orchestra.run.smithi194.stdout:(8/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_64 21 MB/s | 5.3 MB 00:00 2024-09-16T04:04:47.371 INFO:teuthology.orchestra.run.smithi194.stdout:(9/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 1.6 MB/s | 161 kB 00:00 2024-09-16T04:04:47.388 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T04:04:47.396 INFO:teuthology.orchestra.run.smithi194.stdout:(10/32): python3-rados-19.1.1-326.g8921c8ef.el9 6.2 MB/s | 320 kB 00:00 2024-09-16T04:04:47.422 INFO:teuthology.orchestra.run.smithi194.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 5.8 MB/s | 300 kB 00:00 2024-09-16T04:04:47.447 INFO:teuthology.orchestra.run.smithi194.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-16T04:04:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:04:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:47.604 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T04:04:47.647 INFO:teuthology.orchestra.run.smithi194.stdout:(13/32): ceph-common-19.1.1-326.g8921c8ef.el9.x 16 MB/s | 21 MB 00:01 2024-09-16T04:04:47.715 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T04:04:47.730 INFO:teuthology.orchestra.run.smithi194.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 545 kB/s | 45 kB 00:00 2024-09-16T04:04:47.746 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T04:04:47.781 INFO:teuthology.orchestra.run.smithi194.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 298 kB/s | 107 kB 00:00 2024-09-16T04:04:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T04:04:47.806 INFO:teuthology.orchestra.run.smithi194.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-09-16T04:04:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T04:04:47.848 INFO:teuthology.orchestra.run.smithi194.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-16T04:04:47.857 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T04:04:47.882 INFO:teuthology.orchestra.run.smithi194.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-16T04:04:47.890 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T04:04:47.907 INFO:teuthology.orchestra.run.smithi194.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 420 kB/s | 42 kB 00:00 2024-09-16T04:04:47.932 INFO:teuthology.orchestra.run.smithi194.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T04:04:47.958 INFO:teuthology.orchestra.run.smithi194.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 474 kB/s | 36 kB 00:00 2024-09-16T04:04:47.966 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-16T04:04:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T04:04:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T04:04:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T04:04:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T04:04:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T04:04:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T04:04:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T04:04:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T04:04:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T04:04:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T04:04:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:04:47.984 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:04:47.985 INFO:teuthology.orchestra.run.smithi174.stdout:Install 30 Packages 2024-09-16T04:04:47.985 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 2 Packages 2024-09-16T04:04:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:04:47.985 INFO:teuthology.orchestra.run.smithi194.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-16T04:04:47.986 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 60 M 2024-09-16T04:04:47.986 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:04:48.041 INFO:teuthology.orchestra.run.smithi194.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-16T04:04:48.067 INFO:teuthology.orchestra.run.smithi194.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 373 kB/s | 26 kB 00:00 2024-09-16T04:04:48.092 INFO:teuthology.orchestra.run.smithi194.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-09-16T04:04:48.118 INFO:teuthology.orchestra.run.smithi194.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T04:04:48.159 INFO:teuthology.orchestra.run.smithi194.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T04:04:48.185 INFO:teuthology.orchestra.run.smithi194.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-16T04:04:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T04:04:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:48.277 INFO:teuthology.orchestra.run.smithi194.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-16T04:04:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:48.385 INFO:teuthology.orchestra.run.smithi194.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-16T04:04:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T04:04:48.443 INFO:teuthology.orchestra.run.smithi194.stdout:(31/32): librados2-19.1.1-326.g8921c8ef.el9.x86 13 MB/s | 3.4 MB 00:00 2024-09-16T04:04:48.444 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T04:04:48.535 INFO:teuthology.orchestra.run.smithi194.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-16T04:04:48.540 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:48.540 INFO:teuthology.orchestra.run.smithi194.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-16T04:04:48.642 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:04:48.681 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:04:48.681 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:04:48.749 INFO:teuthology.orchestra.run.smithi174.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 9.5 MB/s | 5.4 MB 00:00 2024-09-16T04:04:48.808 INFO:teuthology.orchestra.run.smithi174.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 432 kB/s | 25 kB 00:00 2024-09-16T04:04:48.933 INFO:teuthology.orchestra.run.smithi174.stdout:(3/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 14 MB/s | 11 MB 00:00 2024-09-16T04:04:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T04:04:48.966 INFO:teuthology.orchestra.run.smithi174.stdout:(4/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 4.6 MB/s | 751 kB 00:00 2024-09-16T04:04:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:49.000 INFO:teuthology.orchestra.run.smithi174.stdout:(5/32): libradosstriper1-19.1.1-326.g8921c8ef.e 7.4 MB/s | 506 kB 00:00 2024-09-16T04:04:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:49.059 INFO:teuthology.orchestra.run.smithi174.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 775 kB/s | 45 kB 00:00 2024-09-16T04:04:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T04:04:49.133 INFO:teuthology.orchestra.run.smithi174.stdout:(7/32): python3-ceph-common-19.1.1-326.g8921c8e 1.8 MB/s | 136 kB 00:00 2024-09-16T04:04:49.166 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T04:04:49.201 INFO:teuthology.orchestra.run.smithi174.stdout:(8/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 2.4 MB/s | 161 kB 00:00 2024-09-16T04:04:49.276 INFO:teuthology.orchestra.run.smithi174.stdout:(9/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_64 17 MB/s | 5.3 MB 00:00 2024-09-16T04:04:49.310 INFO:teuthology.orchestra.run.smithi174.stdout:(10/32): python3-rados-19.1.1-326.g8921c8ef.el9 2.9 MB/s | 320 kB 00:00 2024-09-16T04:04:49.329 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:04:49.329 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:04:49.335 INFO:teuthology.orchestra.run.smithi174.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 5.0 MB/s | 300 kB 00:00 2024-09-16T04:04:49.360 INFO:teuthology.orchestra.run.smithi174.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-16T04:04:49.560 INFO:teuthology.orchestra.run.smithi174.stdout:(13/32): ceph-common-19.1.1-326.g8921c8ef.el9.x 15 MB/s | 21 MB 00:01 2024-09-16T04:04:49.669 INFO:teuthology.orchestra.run.smithi174.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-09-16T04:04:49.694 INFO:teuthology.orchestra.run.smithi174.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 297 kB/s | 107 kB 00:00 2024-09-16T04:04:49.719 INFO:teuthology.orchestra.run.smithi174.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-09-16T04:04:49.770 INFO:teuthology.orchestra.run.smithi174.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 849 kB/s | 42 kB 00:00 2024-09-16T04:04:49.795 INFO:teuthology.orchestra.run.smithi174.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-16T04:04:49.845 INFO:teuthology.orchestra.run.smithi174.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 718 kB/s | 36 kB 00:00 2024-09-16T04:04:49.870 INFO:teuthology.orchestra.run.smithi174.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-16T04:04:49.904 INFO:teuthology.orchestra.run.smithi174.stdout:(21/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-09-16T04:04:49.929 INFO:teuthology.orchestra.run.smithi174.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-09-16T04:04:49.988 INFO:teuthology.orchestra.run.smithi174.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-16T04:04:50.013 INFO:teuthology.orchestra.run.smithi174.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 372 kB/s | 26 kB 00:00 2024-09-16T04:04:50.038 INFO:teuthology.orchestra.run.smithi174.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-09-16T04:04:50.064 INFO:teuthology.orchestra.run.smithi174.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T04:04:50.114 INFO:teuthology.orchestra.run.smithi174.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-09-16T04:04:50.139 INFO:teuthology.orchestra.run.smithi174.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-16T04:04:50.239 INFO:teuthology.orchestra.run.smithi174.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-16T04:04:50.357 INFO:teuthology.orchestra.run.smithi174.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-09-16T04:04:50.416 INFO:teuthology.orchestra.run.smithi174.stdout:(31/32): librados2-19.1.1-326.g8921c8ef.el9.x86 12 MB/s | 3.4 MB 00:00 2024-09-16T04:04:50.491 INFO:teuthology.orchestra.run.smithi174.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 13 MB/s | 3.2 MB 00:00 2024-09-16T04:04:50.496 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:50.497 INFO:teuthology.orchestra.run.smithi174.stdout:Total 24 MB/s | 60 MB 00:02 2024-09-16T04:04:50.591 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:04:50.627 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:04:50.627 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:04:50.806 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:04:50.889 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T04:04:50.932 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T04:04:50.965 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T04:04:50.988 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:51.009 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:51.199 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:51.252 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:51.289 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:04:51.290 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:04:51.621 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:51.624 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T04:04:51.672 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:51.719 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:51.834 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:51.931 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T04:04:51.960 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:52.014 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:52.043 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T04:04:52.097 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T04:04:52.163 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:04:52.187 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:52.243 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:52.291 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:52.334 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:52.340 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T04:04:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:52.426 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T04:04:52.426 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T04:04:52.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:04:52.460 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:04:52.579 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T04:04:52.611 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T04:04:52.653 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T04:04:52.685 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T04:04:52.714 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T04:04:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T04:04:52.782 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:04:52.858 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T04:04:52.887 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T04:04:52.912 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T04:04:52.935 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:52.960 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:53.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T04:04:53.118 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:53.145 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:53.198 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T04:04:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T04:04:53.577 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T04:04:53.638 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:53.722 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T04:04:53.787 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:53.832 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T04:04:53.857 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:53.857 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:53.920 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:53.926 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T04:04:53.956 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T04:04:53.957 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T04:04:54.007 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T04:04:54.078 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:04:54.109 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:54.173 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:54.207 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T04:04:54.317 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T04:04:54.350 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T04:04:54.390 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T04:04:54.431 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T04:04:54.460 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T04:04:54.494 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T04:04:54.804 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T04:04:54.847 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:54.920 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:54.969 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T04:04:55.013 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T04:04:55.495 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T04:04:55.541 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:55.613 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:55.689 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T04:04:55.719 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T04:04:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:56.310 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:56.887 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:56.938 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:57.031 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:57.031 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T04:04:57.031 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T04:04:57.031 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:04:57.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:04:57.980 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:58.023 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:58.591 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:58.643 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:58.754 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:58.754 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T04:04:58.754 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T04:04:58.754 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:04:58.786 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:05:03.115 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:05:03.115 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-09-16T04:05:03.115 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-09-16T04:05:03.115 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-09-16T04:05:03.116 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-09-16T04:05:03.116 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-09-16T04:05:03.116 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-09-16T04:05:03.116 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-09-16T04:05:03.116 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:04.131 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:05:04.158 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:05:04.158 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:04.158 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T04:05:04.158 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:05:04.158 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:05:04.158 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:04.160 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:05:04.234 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:05:04.234 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:05:05.386 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:05:05.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T04:05:05.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T04:05:05.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T04:05:05.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T04:05:05.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T04:05:05.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T04:05:05.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T04:05:05.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T04:05:05.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T04:05:05.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T04:05:05.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T04:05:05.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:05:05.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T04:05:05.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T04:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-09-16T04:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T04:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T04:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T04:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T04:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:05:06.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-test 2024-09-16T04:05:06.933 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:05:06.933 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-09-16T04:05:06.933 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-09-16T04:05:06.933 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-09-16T04:05:06.933 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-09-16T04:05:06.934 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-09-16T04:05:06.934 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-09-16T04:05:06.934 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-09-16T04:05:06.934 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:07.495 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:29 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:05:07.644 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:05:07.645 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:07.646 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-09-16T04:05:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:07.646 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 49 M 2024-09-16T04:05:07.646 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 209 M 2024-09-16T04:05:07.646 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:05:07.824 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-16T04:05:07.865 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T04:05:07.932 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 717 kB/s | 163 kB 00:00 2024-09-16T04:05:07.937 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:05:07.963 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:05:07.963 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:07.963 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T04:05:07.963 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:05:07.964 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:05:07.964 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:07.965 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:05:08.056 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:05:08.056 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:05:09.099 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:05:09.100 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-09-16T04:05:09.100 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-09-16T04:05:09.100 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-09-16T04:05:09.100 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-09-16T04:05:09.100 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-09-16T04:05:09.100 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-09-16T04:05:09.100 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-09-16T04:05:09.100 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:09.207 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-16T04:05:09.210 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:09.210 INFO:teuthology.orchestra.run.smithi016.stdout:Total 31 MB/s | 49 MB 00:01 2024-09-16T04:05:09.238 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T04:05:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T04:05:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T04:05:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T04:05:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T04:05:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T04:05:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T04:05:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T04:05:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T04:05:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:05:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T04:05:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T04:05:09.260 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:05:09.260 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:05:09.677 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:05:09.678 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:05:10.066 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:05:10.091 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:05:10.091 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:10.091 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T04:05:10.091 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:05:10.091 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:05:10.091 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:10.093 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:05:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:05:10.177 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:05:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:05:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T04:05:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T04:05:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:05:10.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:05:10.539 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:10.539 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-09-16T04:05:10.539 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.539 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.539 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:05:10.539 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T04:05:10.539 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.540 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.540 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.540 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.540 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T04:05:10.540 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T04:05:10.540 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T04:05:10.541 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.541 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T04:05:10.541 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T04:05:10.541 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T04:05:10.541 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.541 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T04:05:10.542 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.542 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T04:05:10.542 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T04:05:10.542 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T04:05:10.542 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.542 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T04:05:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T04:05:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T04:05:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T04:05:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T04:05:10.544 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T04:05:10.544 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:10.544 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:05:10.911 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-09-16T04:05:11.338 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:05:11.338 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T04:05:11.339 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T04:05:11.339 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T04:05:11.339 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T04:05:11.339 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T04:05:11.339 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T04:05:11.339 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:05:11.339 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T04:05:11.340 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T04:05:11.340 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T04:05:11.340 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T04:05:11.340 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:05:11.340 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:05:11.340 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T04:05:11.340 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T04:05:11.340 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T04:05:11.340 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T04:05:11.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T04:05:11.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T04:05:11.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T04:05:11.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T04:05:11.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T04:05:11.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T04:05:11.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T04:05:11.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T04:05:11.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T04:05:11.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T04:05:11.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T04:05:11.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T04:05:11.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T04:05:11.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:05:11.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T04:05:11.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T04:05:11.490 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:05:11.622 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:05:11.622 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:11.622 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:05:11.622 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:11.622 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:05:11.622 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T04:05:11.623 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-16T04:05:11.623 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T04:05:11.623 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T04:05:11.623 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T04:05:11.623 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:11.623 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:05:11.623 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:11.623 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-09-16T04:05:11.623 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:11.624 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 49 M 2024-09-16T04:05:11.624 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 209 M 2024-09-16T04:05:11.624 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:05:11.945 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.6 MB/s | 304 kB 00:00 2024-09-16T04:05:11.970 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 770 kB/s | 163 kB 00:00 2024-09-16T04:05:11.986 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T04:05:12.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:05:12.481 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:12.481 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T04:05:12.482 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:12.484 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T04:05:12.484 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T04:05:12.484 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T04:05:12.484 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T04:05:12.484 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T04:05:12.484 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:12.484 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:05:12.822 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-test 2024-09-16T04:05:13.237 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 33 MB/s | 49 MB 00:01 2024-09-16T04:05:13.239 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:13.239 INFO:teuthology.orchestra.run.smithi194.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-16T04:05:13.268 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:05:13.287 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:05:13.287 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:05:13.382 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:05:13.516 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:05:13.516 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:05:13.517 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:13.517 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:05:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T04:05:13.517 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-16T04:05:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T04:05:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T04:05:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T04:05:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:13.518 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:05:13.518 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:13.518 INFO:teuthology.orchestra.run.smithi174.stdout:Install 4 Packages 2024-09-16T04:05:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:13.519 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 49 M 2024-09-16T04:05:13.519 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 209 M 2024-09-16T04:05:13.519 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:05:13.682 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:05:13.682 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:05:13.782 INFO:teuthology.orchestra.run.smithi174.stdout:(1/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 757 kB/s | 163 kB 00:00 2024-09-16T04:05:14.157 INFO:teuthology.orchestra.run.smithi174.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-09-16T04:05:14.199 INFO:teuthology.orchestra.run.smithi174.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 481 kB/s | 304 kB 00:00 2024-09-16T04:05:14.242 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:05:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T04:05:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T04:05:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:05:15.091 INFO:teuthology.orchestra.run.smithi174.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-16T04:05:15.092 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:15.093 INFO:teuthology.orchestra.run.smithi174.stdout:Total 31 MB/s | 49 MB 00:01 2024-09-16T04:05:15.119 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:05:15.139 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:05:15.139 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:05:15.290 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:05:15.369 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:05:15.524 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:05:15.525 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:05:16.051 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:05:16.151 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T04:05:16.197 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T04:05:16.225 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:05:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:05:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T04:05:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T04:05:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T04:05:18.097 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T04:05:18.097 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:18.097 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:05:18.098 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:18.098 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:18.098 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T04:05:18.098 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T04:05:18.098 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:18.098 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:05:18.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph 2024-09-16T04:05:18.951 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:05:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:05:19.104 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:05:19.107 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:19.107 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:05:19.107 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:19.107 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:05:19.107 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T04:05:19.108 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T04:05:19.109 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T04:05:19.110 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout:Install 37 Packages 2024-09-16T04:05:19.111 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:05:19.112 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 30 M 2024-09-16T04:05:19.113 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 108 M 2024-09-16T04:05:19.113 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:05:19.694 INFO:teuthology.orchestra.run.smithi016.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 57 kB/s | 6.5 kB 00:00 2024-09-16T04:05:19.935 INFO:teuthology.orchestra.run.smithi016.stdout:(2/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.7 MB/s | 2.4 MB 00:00 2024-09-16T04:05:19.977 INFO:teuthology.orchestra.run.smithi016.stdout:(3/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 3.8 MB/s | 1.5 MB 00:00 2024-09-16T04:05:20.069 INFO:teuthology.orchestra.run.smithi016.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 2.7 MB/s | 249 kB 00:00 2024-09-16T04:05:20.136 INFO:teuthology.orchestra.run.smithi016.stdout:(5/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-16T04:05:20.253 INFO:teuthology.orchestra.run.smithi016.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-16T04:05:20.295 INFO:teuthology.orchestra.run.smithi016.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T04:05:20.337 INFO:teuthology.orchestra.run.smithi016.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 6.1 MB/s | 1.2 MB 00:00 2024-09-16T04:05:20.363 INFO:teuthology.orchestra.run.smithi016.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-16T04:05:20.388 INFO:teuthology.orchestra.run.smithi016.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-16T04:05:20.422 INFO:teuthology.orchestra.run.smithi016.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T04:05:20.597 INFO:teuthology.orchestra.run.smithi016.stdout:(12/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_ 25 MB/s | 17 MB 00:00 2024-09-16T04:05:20.648 INFO:teuthology.orchestra.run.smithi016.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 727 kB/s | 188 kB 00:00 2024-09-16T04:05:20.673 INFO:teuthology.orchestra.run.smithi016.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 462 kB/s | 35 kB 00:00 2024-09-16T04:05:20.707 INFO:teuthology.orchestra.run.smithi016.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 605 kB/s | 172 kB 00:00 2024-09-16T04:05:20.731 INFO:teuthology.orchestra.run.smithi016.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 922 kB/s | 77 kB 00:00 2024-09-16T04:05:20.757 INFO:teuthology.orchestra.run.smithi016.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 609 kB/s | 51 kB 00:00 2024-09-16T04:05:20.782 INFO:teuthology.orchestra.run.smithi016.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-16T04:05:20.818 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:05:20.832 INFO:teuthology.orchestra.run.smithi016.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 221 kB/s | 22 kB 00:00 2024-09-16T04:05:20.857 INFO:teuthology.orchestra.run.smithi016.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 577 kB/s | 43 kB 00:00 2024-09-16T04:05:20.891 INFO:teuthology.orchestra.run.smithi016.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-09-16T04:05:20.914 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:05:20.917 INFO:teuthology.orchestra.run.smithi016.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 466 kB/s | 11 kB 00:00 2024-09-16T04:05:20.942 INFO:teuthology.orchestra.run.smithi016.stdout:(23/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.6 MB/s | 358 kB 00:00 2024-09-16T04:05:20.967 INFO:teuthology.orchestra.run.smithi016.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 350 kB/s | 18 kB 00:00 2024-09-16T04:05:20.993 INFO:teuthology.orchestra.run.smithi016.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-09-16T04:05:21.018 INFO:teuthology.orchestra.run.smithi016.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-16T04:05:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:(27/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 816 kB/s | 172 kB 00:00 2024-09-16T04:05:21.068 INFO:teuthology.orchestra.run.smithi016.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-16T04:05:21.094 INFO:teuthology.orchestra.run.smithi016.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-09-16T04:05:21.119 INFO:teuthology.orchestra.run.smithi016.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T04:05:21.144 INFO:teuthology.orchestra.run.smithi016.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-16T04:05:21.169 INFO:teuthology.orchestra.run.smithi016.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-16T04:05:21.195 INFO:teuthology.orchestra.run.smithi016.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-16T04:05:21.220 INFO:teuthology.orchestra.run.smithi016.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-16T04:05:21.246 INFO:teuthology.orchestra.run.smithi016.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-16T04:05:21.262 INFO:teuthology.orchestra.run.smithi016.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 477 kB/s | 20 kB 00:00 2024-09-16T04:05:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-16T04:05:21.318 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:21.318 INFO:teuthology.orchestra.run.smithi016.stdout:Total 14 MB/s | 30 MB 00:02 2024-09-16T04:05:21.476 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:05:21.550 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:05:21.550 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T04:05:21.550 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T04:05:21.550 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:05:22.055 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:05:22.055 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:05:22.170 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-09-16T04:05:22.751 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:39 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:05:22.902 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T04:05:22.905 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T04:05:22.906 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T04:05:22.907 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout:Install 37 Packages 2024-09-16T04:05:22.908 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:22.909 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 30 M 2024-09-16T04:05:22.909 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 108 M 2024-09-16T04:05:22.909 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:05:23.313 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:05:23.313 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T04:05:23.313 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T04:05:23.313 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T04:05:23.610 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:05:23.648 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T04:05:23.648 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:23.648 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:05:23.648 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:23.648 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:23.648 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T04:05:23.648 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T04:05:23.648 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:23.648 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:05:23.678 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T04:05:23.721 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T04:05:23.755 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T04:05:23.824 INFO:teuthology.orchestra.run.smithi194.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 47 kB/s | 6.5 kB 00:00 2024-09-16T04:05:23.831 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T04:05:23.872 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T04:05:23.904 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T04:05:23.926 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph 2024-09-16T04:05:23.991 INFO:teuthology.orchestra.run.smithi194.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 5.0 MB/s | 1.5 MB 00:00 2024-09-16T04:05:24.001 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T04:05:24.041 INFO:teuthology.orchestra.run.smithi194.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.7 MB/s | 2.4 MB 00:00 2024-09-16T04:05:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T04:05:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T04:05:24.128 INFO:teuthology.orchestra.run.smithi194.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 2.9 MB/s | 249 kB 00:00 2024-09-16T04:05:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T04:05:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T04:05:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T04:05:24.241 INFO:teuthology.orchestra.run.smithi194.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-16T04:05:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T04:05:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T04:05:24.325 INFO:teuthology.orchestra.run.smithi194.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-16T04:05:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T04:05:24.367 INFO:teuthology.orchestra.run.smithi194.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T04:05:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T04:05:24.433 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T04:05:24.473 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T04:05:24.521 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:39 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:05:24.534 INFO:teuthology.orchestra.run.smithi194.stdout:(8/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 31 MB/s | 17 MB 00:00 2024-09-16T04:05:24.548 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T04:05:24.559 INFO:teuthology.orchestra.run.smithi194.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 704 kB/s | 135 kB 00:00 2024-09-16T04:05:24.626 INFO:teuthology.orchestra.run.smithi194.stdout:(10/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 5.9 MB/s | 4.7 MB 00:00 2024-09-16T04:05:24.651 INFO:teuthology.orchestra.run.smithi194.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-16T04:05:24.666 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:05:24.669 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:24.669 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:05:24.669 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:24.669 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:05:24.669 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T04:05:24.669 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T04:05:24.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T04:05:24.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout:Install 37 Packages 2024-09-16T04:05:24.672 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:24.673 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 30 M 2024-09-16T04:05:24.673 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 108 M 2024-09-16T04:05:24.673 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:05:24.685 INFO:teuthology.orchestra.run.smithi194.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 831 kB/s | 125 kB 00:00 2024-09-16T04:05:24.718 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T04:05:24.768 INFO:teuthology.orchestra.run.smithi194.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-09-16T04:05:24.794 INFO:teuthology.orchestra.run.smithi194.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-16T04:05:24.811 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T04:05:24.819 INFO:teuthology.orchestra.run.smithi194.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-16T04:05:24.844 INFO:teuthology.orchestra.run.smithi194.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 894 kB/s | 172 kB 00:00 2024-09-16T04:05:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T04:05:24.869 INFO:teuthology.orchestra.run.smithi194.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-16T04:05:24.894 INFO:teuthology.orchestra.run.smithi194.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-09-16T04:05:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T04:05:24.954 INFO:teuthology.orchestra.run.smithi194.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 747 kB/s | 43 kB 00:00 2024-09-16T04:05:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T04:05:24.978 INFO:teuthology.orchestra.run.smithi194.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-16T04:05:25.004 INFO:teuthology.orchestra.run.smithi194.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-09-16T04:05:25.029 INFO:teuthology.orchestra.run.smithi194.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-16T04:05:25.036 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T04:05:25.062 INFO:teuthology.orchestra.run.smithi194.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 685 kB/s | 18 kB 00:00 2024-09-16T04:05:25.088 INFO:teuthology.orchestra.run.smithi194.stdout:(24/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-09-16T04:05:25.113 INFO:teuthology.orchestra.run.smithi194.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-09-16T04:05:25.138 INFO:teuthology.orchestra.run.smithi194.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 314 kB/s | 23 kB 00:00 2024-09-16T04:05:25.163 INFO:teuthology.orchestra.run.smithi194.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-09-16T04:05:25.189 INFO:teuthology.orchestra.run.smithi194.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-16T04:05:25.214 INFO:teuthology.orchestra.run.smithi194.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi194.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-16T04:05:25.243 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T04:05:25.265 INFO:teuthology.orchestra.run.smithi194.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-16T04:05:25.290 INFO:teuthology.orchestra.run.smithi194.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-16T04:05:25.297 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T04:05:25.315 INFO:teuthology.orchestra.run.smithi194.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-09-16T04:05:25.340 INFO:teuthology.orchestra.run.smithi194.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-16T04:05:25.377 INFO:teuthology.orchestra.run.smithi174.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 48 kB/s | 6.5 kB 00:00 2024-09-16T04:05:25.377 INFO:teuthology.orchestra.run.smithi194.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-16T04:05:25.378 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T04:05:25.399 INFO:teuthology.orchestra.run.smithi194.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-09-16T04:05:25.424 INFO:teuthology.orchestra.run.smithi194.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T04:05:25.430 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:25.430 INFO:teuthology.orchestra.run.smithi194.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-16T04:05:25.476 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T04:05:25.516 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T04:05:25.584 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:05:25.623 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:05:25.624 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:05:25.643 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T04:05:25.716 INFO:teuthology.orchestra.run.smithi174.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 3.2 MB/s | 1.5 MB 00:00 2024-09-16T04:05:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T04:05:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:05:25.858 INFO:teuthology.orchestra.run.smithi174.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 3.8 MB/s | 2.4 MB 00:00 2024-09-16T04:05:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:05:25.884 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:25.884 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T04:05:25.884 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:05:25.884 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:05:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:25.925 INFO:teuthology.orchestra.run.smithi174.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 3.7 MB/s | 249 kB 00:00 2024-09-16T04:05:26.067 INFO:teuthology.orchestra.run.smithi174.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-16T04:05:26.109 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:05:26.109 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:05:26.167 INFO:teuthology.orchestra.run.smithi174.stdout:(6/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 5.9 MB/s | 4.7 MB 00:00 2024-09-16T04:05:26.212 INFO:teuthology.orchestra.run.smithi174.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 8.5 MB/s | 1.2 MB 00:00 2024-09-16T04:05:26.254 INFO:teuthology.orchestra.run.smithi174.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-16T04:05:26.296 INFO:teuthology.orchestra.run.smithi174.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-16T04:05:26.321 INFO:teuthology.orchestra.run.smithi174.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 692 kB/s | 106 kB 00:00 2024-09-16T04:05:26.355 INFO:teuthology.orchestra.run.smithi174.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T04:05:26.456 INFO:teuthology.orchestra.run.smithi174.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-09-16T04:05:26.481 INFO:teuthology.orchestra.run.smithi174.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-16T04:05:26.506 INFO:teuthology.orchestra.run.smithi174.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 691 kB/s | 35 kB 00:00 2024-09-16T04:05:26.531 INFO:teuthology.orchestra.run.smithi174.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-16T04:05:26.556 INFO:teuthology.orchestra.run.smithi174.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T04:05:26.582 INFO:teuthology.orchestra.run.smithi174.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 829 kB/s | 42 kB 00:00 2024-09-16T04:05:26.641 INFO:teuthology.orchestra.run.smithi174.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-16T04:05:26.666 INFO:teuthology.orchestra.run.smithi174.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-16T04:05:26.682 INFO:teuthology.orchestra.run.smithi174.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-16T04:05:26.716 INFO:teuthology.orchestra.run.smithi174.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-16T04:05:26.733 INFO:teuthology.orchestra.run.smithi174.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 708 kB/s | 11 kB 00:00 2024-09-16T04:05:26.750 INFO:teuthology.orchestra.run.smithi174.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-16T04:05:26.767 INFO:teuthology.orchestra.run.smithi174.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-09-16T04:05:26.792 INFO:teuthology.orchestra.run.smithi174.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-09-16T04:05:26.817 INFO:teuthology.orchestra.run.smithi174.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-16T04:05:26.843 INFO:teuthology.orchestra.run.smithi174.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T04:05:26.868 INFO:teuthology.orchestra.run.smithi174.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-16T04:05:26.893 INFO:teuthology.orchestra.run.smithi174.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T04:05:26.919 INFO:teuthology.orchestra.run.smithi174.stdout:(30/37): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-16T04:05:26.944 INFO:teuthology.orchestra.run.smithi174.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-16T04:05:26.977 INFO:teuthology.orchestra.run.smithi174.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-16T04:05:27.005 INFO:teuthology.orchestra.run.smithi174.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2024-09-16T04:05:27.028 INFO:teuthology.orchestra.run.smithi174.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-16T04:05:27.187 INFO:teuthology.orchestra.run.smithi174.stdout:(35/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_ 11 MB/s | 17 MB 00:01 2024-09-16T04:05:27.212 INFO:teuthology.orchestra.run.smithi174.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 109 kB/s | 20 kB 00:00 2024-09-16T04:05:27.245 INFO:teuthology.orchestra.run.smithi174.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-09-16T04:05:27.249 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:27.249 INFO:teuthology.orchestra.run.smithi174.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-16T04:05:27.399 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:05:27.414 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:05:27.438 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:05:27.438 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:05:27.444 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:05:27.444 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:27.444 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T04:05:27.444 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:05:27.444 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:05:27.444 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:27.481 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:05:27.552 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T04:05:27.586 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T04:05:27.630 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T04:05:27.702 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T04:05:27.745 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T04:05:27.778 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T04:05:27.869 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T04:05:27.904 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:05:27.905 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:05:27.907 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:05:27.933 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:05:27.933 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:27.933 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T04:05:27.933 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:05:27.933 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:05:27.933 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T04:05:27.997 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T04:05:28.032 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T04:05:28.062 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T04:05:28.096 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T04:05:28.145 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T04:05:28.166 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:05:28.186 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T04:05:28.191 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:05:28.191 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:28.191 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T04:05:28.191 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:05:28.192 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:05:28.192 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:28.229 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T04:05:28.257 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:05:28.263 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T04:05:28.311 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T04:05:28.343 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T04:05:28.416 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T04:05:28.554 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T04:05:28.634 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T04:05:28.696 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T04:05:28.734 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T04:05:28.782 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T04:05:28.838 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T04:05:29.024 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T04:05:29.071 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T04:05:29.149 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T04:05:29.237 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T04:05:29.292 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:05:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T04:05:29.357 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T04:05:29.390 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T04:05:29.419 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T04:05:29.426 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T04:05:29.497 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T04:05:29.566 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T04:05:29.574 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T04:05:29.599 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T04:05:29.605 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:05:29.632 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:05:29.632 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:29.632 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T04:05:29.632 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:05:29.633 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:05:29.633 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:29.691 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T04:05:29.777 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T04:05:29.827 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T04:05:29.861 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T04:05:29.891 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T04:05:29.943 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T04:05:29.976 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T04:05:30.016 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T04:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T04:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T04:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T04:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T04:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T04:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T04:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T04:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T04:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T04:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T04:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T04:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T04:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T04:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T04:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T04:05:30.078 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T04:05:30.109 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T04:05:30.158 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T04:05:30.190 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T04:05:30.264 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T04:05:30.413 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T04:05:30.499 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T04:05:30.553 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T04:05:30.591 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T04:05:30.640 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T04:05:30.704 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T04:05:30.891 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T04:05:30.945 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T04:05:30.945 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T04:05:30.945 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:30.945 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:05:30.945 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:30.945 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T04:05:30.946 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T04:05:30.947 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:30.948 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:05:31.023 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T04:05:31.081 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:05:31.109 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:05:31.109 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:31.109 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T04:05:31.109 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:05:31.109 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:05:31.109 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:31.111 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T04:05:31.159 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T04:05:31.278 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T04:05:31.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-base 2024-09-16T04:05:31.449 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T04:05:31.479 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:05:31.505 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:05:31.505 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:31.505 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T04:05:31.505 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:05:31.505 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:05:31.505 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:31.548 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:05:31.573 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:05:31.573 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:31.573 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T04:05:31.573 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:05:31.573 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:05:31.573 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:31.799 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:05:31.823 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:05:31.823 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:31.823 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T04:05:31.823 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:05:31.823 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:05:31.823 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:31.883 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:05:31.886 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:05:31.982 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:32.029 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:05:32.030 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-16T04:05:32.030 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:05:32.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install cephadm 2024-09-16T04:05:32.695 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:05:32.836 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:05:32.836 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 765 k 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 771 k 2024-09-16T04:05:32.837 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:05:32.938 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:05:32.965 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:05:32.966 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:32.966 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T04:05:32.966 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:05:32.966 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:05:32.966 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:33.333 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.5 MB/s | 765 kB 00:00 2024-09-16T04:05:33.333 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:33.334 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.5 MB/s | 765 kB 00:00 2024-09-16T04:05:33.334 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:05:33.340 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:05:33.340 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:05:33.346 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:05:33.346 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:05:33.405 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:05:33.419 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:05:33.429 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:05:33.429 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:33.429 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T04:05:33.430 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:05:33.430 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:05:33.430 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:33.534 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:05:33.534 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T04:05:33.534 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T04:05:33.535 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T04:05:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T04:05:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T04:05:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T04:05:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T04:05:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T04:05:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T04:05:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T04:05:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T04:05:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T04:05:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T04:05:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T04:05:33.656 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:05:33.680 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:05:33.680 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:33.680 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T04:05:33.680 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:05:33.680 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:05:33.680 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:33.753 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:05:33.888 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:33.982 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:34.598 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:05:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:34.599 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:34.599 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:05:34.654 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T04:05:34.654 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:34.654 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:05:34.655 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:34.655 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:34.655 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:34.655 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:34.655 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:34.656 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:34.656 INFO:teuthology.orchestra.run.smithi194.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T04:05:34.656 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T04:05:34.656 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T04:05:34.656 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T04:05:34.657 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T04:05:34.657 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T04:05:34.657 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T04:05:34.657 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T04:05:34.657 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T04:05:34.657 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T04:05:34.658 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T04:05:34.658 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T04:05:34.658 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T04:05:34.658 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T04:05:34.658 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T04:05:34.658 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T04:05:34.659 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T04:05:34.659 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T04:05:34.659 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T04:05:34.659 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T04:05:34.659 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T04:05:34.659 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T04:05:34.660 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T04:05:34.660 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T04:05:34.660 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T04:05:34.660 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T04:05:34.660 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T04:05:34.660 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T04:05:34.661 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T04:05:34.661 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T04:05:34.661 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T04:05:34.661 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:34.661 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:05:34.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T04:05:35.048 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-09-16T04:05:35.379 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:05:35.518 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:05:35.518 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:35.518 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-09-16T04:05:35.519 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:35.519 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:05:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T04:05:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:35.519 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:05:35.519 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:35.519 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-16T04:05:35.519 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:35.520 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 145 k 2024-09-16T04:05:35.520 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 435 k 2024-09-16T04:05:35.520 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:05:35.589 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T04:05:35.590 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T04:05:35.591 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T04:05:35.592 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T04:05:35.592 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T04:05:35.592 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T04:05:35.592 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T04:05:35.592 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T04:05:35.592 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T04:05:35.592 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T04:05:35.592 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T04:05:35.612 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:05:35.702 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:35.734 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 678 kB/s | 145 kB 00:00 2024-09-16T04:05:35.734 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:35.735 INFO:teuthology.orchestra.run.smithi016.stdout:Total 673 kB/s | 145 kB 00:00 2024-09-16T04:05:35.735 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:05:35.743 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:05:35.743 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:05:35.744 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:05:35.745 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-16T04:05:35.745 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:05:35.804 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:05:35.804 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:05:35.831 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install cephadm 2024-09-16T04:05:35.963 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:05:36.052 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:05:36.078 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:05:36.078 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:36.078 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T04:05:36.078 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:36.398 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:05:36.530 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:05:36.531 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:36.531 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:05:36.531 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:36.531 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:05:36.531 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T04:05:36.531 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:36.531 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:05:36.531 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:36.531 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-16T04:05:36.532 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:36.532 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 765 k 2024-09-16T04:05:36.532 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 771 k 2024-09-16T04:05:36.532 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:05:36.633 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T04:05:36.633 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:36.633 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:05:36.633 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:36.633 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:36.634 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:36.634 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:36.634 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:36.634 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:36.634 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T04:05:36.634 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T04:05:36.634 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T04:05:36.634 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T04:05:36.635 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T04:05:36.636 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T04:05:36.637 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T04:05:36.637 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T04:05:36.637 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T04:05:36.637 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T04:05:36.637 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:36.637 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:05:36.792 INFO:teuthology.orchestra.run.smithi194.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-16T04:05:36.793 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:36.793 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-16T04:05:36.793 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:05:36.798 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:05:36.798 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:05:36.804 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:05:36.804 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:05:36.898 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:05:36.974 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:05:36.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:36.975 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:05:36.975 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:36.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:36.975 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:05:36.996 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-base 2024-09-16T04:05:37.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr 2024-09-16T04:05:37.378 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:37.461 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:37.565 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:05:37.656 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:37.697 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:05:37.698 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-16T04:05:37.698 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:05:37.759 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install cephadm 2024-09-16T04:05:37.793 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:05:37.847 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:37.886 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:37.930 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:05:37.931 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-16T04:05:37.932 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:05:38.022 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T04:05:38.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:38.114 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:38.114 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:05:38.114 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:38.114 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:38.114 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:05:38.320 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:05:38.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T04:05:38.453 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-16T04:05:38.454 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:38.455 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 765 k 2024-09-16T04:05:38.455 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 771 k 2024-09-16T04:05:38.455 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:05:38.612 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:05:38.729 INFO:teuthology.orchestra.run.smithi174.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.7 MB/s | 765 kB 00:00 2024-09-16T04:05:38.730 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:38.730 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.7 MB/s | 765 kB 00:00 2024-09-16T04:05:38.731 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:05:38.736 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:05:38.736 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:05:38.742 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:05:38.742 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:05:38.751 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T04:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-16T04:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-09-16T04:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 8.7 M 2024-09-16T04:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 104 M 2024-09-16T04:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:05:38.815 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:05:38.925 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:05:39.062 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 145 k 2024-09-16T04:05:39.063 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 435 k 2024-09-16T04:05:39.064 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:05:39.222 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:39.263 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 222 kB/s | 32 kB 00:00 2024-09-16T04:05:39.264 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 727 kB/s | 145 kB 00:00 2024-09-16T04:05:39.264 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:39.264 INFO:teuthology.orchestra.run.smithi194.stdout:Total 721 kB/s | 145 kB 00:00 2024-09-16T04:05:39.265 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:05:39.272 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:05:39.272 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:05:39.280 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 102 kB/s | 16 kB 00:00 2024-09-16T04:05:39.308 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:39.329 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:05:39.329 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:05:39.488 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:05:39.488 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 9.6 MB/s | 3.6 MB 00:00 2024-09-16T04:05:39.539 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 269 kB 00:00 2024-09-16T04:05:39.565 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:05:39.589 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:05:39.590 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:39.590 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T04:05:39.590 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:39.606 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.1 MB/s | 548 kB 00:00 2024-09-16T04:05:39.631 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 778 kB/s | 19 kB 00:00 2024-09-16T04:05:39.705 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:39.723 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-09-16T04:05:39.765 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-09-16T04:05:39.799 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T04:05:39.832 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.9 MB/s | 188 kB 00:00 2024-09-16T04:05:39.866 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.7 MB/s | 1.0 MB 00:00 2024-09-16T04:05:39.900 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 819 kB/s | 144 kB 00:00 2024-09-16T04:05:39.904 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:39.904 INFO:teuthology.orchestra.run.smithi016.stdout:Total 7.5 MB/s | 8.7 MB 00:01 2024-09-16T04:05:39.954 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:05:39.973 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:39.973 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:39.973 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:05:39.973 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:39.973 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:39.973 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:05:39.974 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:05:39.974 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:05:40.191 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:05:40.191 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:05:40.217 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T04:05:40.523 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:05:40.523 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:40.523 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:05:40.523 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:40.523 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:40.523 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:05:40.738 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:05:40.748 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-09-16T04:05:40.795 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:05:40.818 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T04:05:40.861 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T04:05:40.909 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T04:05:40.929 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-16T04:05:40.930 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:40.931 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 145 k 2024-09-16T04:05:40.932 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 435 k 2024-09-16T04:05:40.932 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:05:40.938 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T04:05:41.030 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T04:05:41.135 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 712 kB/s | 145 kB 00:00 2024-09-16T04:05:41.136 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:41.136 INFO:teuthology.orchestra.run.smithi174.stdout:Total 706 kB/s | 145 kB 00:00 2024-09-16T04:05:41.136 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:05:41.144 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:05:41.144 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:05:41.170 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T04:05:41.201 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:05:41.201 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:05:41.262 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T04:05:41.311 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T04:05:41.326 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:05:41.338 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:05:41.355 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:05:41.370 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T04:05:41.419 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:41.461 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:05:41.462 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-16T04:05:41.462 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:05:41.478 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:05:41.504 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:05:41.504 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:41.504 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T04:05:41.504 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:41.542 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T04:05:42.106 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:05:42.241 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:05:42.241 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T04:05:42.242 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T04:05:42.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T04:05:42.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T04:05:42.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T04:05:42.243 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:42.243 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:05:42.243 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:05:42.243 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-09-16T04:05:42.243 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:42.244 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 8.7 M 2024-09-16T04:05:42.244 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 104 M 2024-09-16T04:05:42.244 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:05:42.444 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:05:42.444 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:42.444 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:05:42.444 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:42.444 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:42.444 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:05:42.486 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T04:05:42.521 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:05:42.550 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 197 kB/s | 32 kB 00:00 2024-09-16T04:05:42.575 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 88 kB/s | 16 kB 00:00 2024-09-16T04:05:42.706 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr 2024-09-16T04:05:42.784 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 9.0 MB/s | 3.6 MB 00:00 2024-09-16T04:05:42.992 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-09-16T04:05:43.051 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T04:05:43.068 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-09-16T04:05:43.151 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 24 MB/s | 2.0 MB 00:00 2024-09-16T04:05:43.185 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-16T04:05:43.210 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T04:05:43.236 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-16T04:05:43.275 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:05:43.369 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:43.411 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:05:43.412 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-16T04:05:43.412 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:05:43.436 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): python3-protobuf-3.14.0-13.el9.noarch. 313 kB/s | 269 kB 00:00 2024-09-16T04:05:43.475 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T04:05:43.486 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-09-16T04:05:43.490 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:43.490 INFO:teuthology.orchestra.run.smithi194.stdout:Total 7.0 MB/s | 8.7 MB 00:01 2024-09-16T04:05:43.536 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:05:43.553 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:05:43.553 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:05:43.769 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:05:43.769 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:05:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:05:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T04:05:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T04:05:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T04:05:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T04:05:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T04:05:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T04:05:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T04:05:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T04:05:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:05:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T04:05:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T04:05:44.038 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:05:44.170 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:05:44.171 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:44.171 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:05:44.171 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:44.171 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:05:44.171 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T04:05:44.171 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:44.172 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:05:44.173 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:05:44.173 INFO:teuthology.orchestra.run.smithi174.stdout:Install 12 Packages 2024-09-16T04:05:44.173 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:44.173 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 8.7 M 2024-09-16T04:05:44.173 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 104 M 2024-09-16T04:05:44.174 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:05:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T04:05:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:44.248 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:44.249 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:05:44.298 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:05:44.386 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T04:05:44.422 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T04:05:44.469 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T04:05:44.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T04:05:44.499 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T04:05:44.591 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T04:05:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T04:05:44.817 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T04:05:44.861 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T04:05:44.880 INFO:teuthology.orchestra.run.smithi174.stdout:(1/12): ceph-prometheus-alerts-19.1.1-326.g8921 81 kB/s | 16 kB 00:00 2024-09-16T04:05:44.890 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:05:44.906 INFO:teuthology.orchestra.run.smithi174.stdout:(2/12): ceph-grafana-dashboards-19.1.1-326.g892 139 kB/s | 32 kB 00:00 2024-09-16T04:05:44.921 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T04:05:45.040 INFO:teuthology.orchestra.run.smithi174.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 269 kB 00:00 2024-09-16T04:05:45.064 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:07 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:05:45.090 INFO:teuthology.orchestra.run.smithi174.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-09-16T04:05:45.148 INFO:teuthology.orchestra.run.smithi174.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T04:05:45.173 INFO:teuthology.orchestra.run.smithi174.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-09-16T04:05:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T04:05:45.207 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T04:05:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T04:05:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T04:05:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T04:05:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T04:05:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:45.208 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:05:45.208 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-09-16T04:05:45.208 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-09-16T04:05:45.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:45.209 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 38 M 2024-09-16T04:05:45.209 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 212 M 2024-09-16T04:05:45.209 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:05:45.274 INFO:teuthology.orchestra.run.smithi174.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 20 MB/s | 2.0 MB 00:00 2024-09-16T04:05:45.299 INFO:teuthology.orchestra.run.smithi174.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-16T04:05:45.325 INFO:teuthology.orchestra.run.smithi174.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T04:05:45.358 INFO:teuthology.orchestra.run.smithi174.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2024-09-16T04:05:45.475 INFO:teuthology.orchestra.run.smithi174.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.9 MB/s | 863 kB 00:00 2024-09-16T04:05:45.759 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-16T04:05:45.801 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.2 MB/s | 801 kB 00:00 2024-09-16T04:05:45.850 INFO:teuthology.orchestra.run.smithi174.stdout:(12/12): ceph-mgr-dashboard-19.1.1-326.g8921c8e 3.0 MB/s | 3.6 MB 00:01 2024-09-16T04:05:45.854 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:45.854 INFO:teuthology.orchestra.run.smithi174.stdout:Total 5.2 MB/s | 8.7 MB 00:01 2024-09-16T04:05:45.907 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:05:45.918 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 207 kB/s | 33 kB 00:00 2024-09-16T04:05:45.925 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:05:45.925 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:05:45.976 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T04:05:45.985 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 274 kB/s | 18 kB 00:00 2024-09-16T04:05:46.014 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:05:46.068 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 462 kB/s | 38 kB 00:00 2024-09-16T04:05:46.132 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:05:46.132 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:05:46.218 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.2 MB/s | 3.0 MB 00:00 2024-09-16T04:05:46.294 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-09-16T04:05:46.452 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2024-09-16T04:05:46.528 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 5.1 MB 00:00 2024-09-16T04:05:46.553 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-16T04:05:46.661 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:05:46.712 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): ceph-mgr-diskprediction-local-19.1.1-3 6.7 MB/s | 7.4 MB 00:01 2024-09-16T04:05:46.751 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T04:05:46.794 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T04:05:46.841 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T04:05:46.880 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T04:05:46.977 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T04:05:46.987 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-16T04:05:46.989 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:46.989 INFO:teuthology.orchestra.run.smithi016.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-16T04:05:47.113 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T04:05:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:05:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T04:05:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T04:05:47.176 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T04:05:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T04:05:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T04:05:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T04:05:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T04:05:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T04:05:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:05:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T04:05:47.177 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T04:05:47.199 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T04:05:47.242 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T04:05:47.271 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:05:47.303 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T04:05:47.363 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T04:05:47.494 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T04:05:47.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T04:05:47.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T04:05:47.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T04:05:47.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T04:05:47.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T04:05:47.495 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:47.495 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:05:47.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T04:05:47.850 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:05:47.850 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:05:48.325 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:05:48.350 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T04:05:48.386 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:05:48.460 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout:====================================================================================== 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout:====================================================================================== 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T04:05:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout:====================================================================================== 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-09-16T04:05:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:05:48.463 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 38 M 2024-09-16T04:05:48.463 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 212 M 2024-09-16T04:05:48.463 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:05:48.637 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:05:48.718 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T04:05:48.778 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:05:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:05:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T04:05:49.043 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-16T04:05:49.077 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-16T04:05:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T04:05:49.152 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T04:05:49.160 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-16T04:05:49.232 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T04:05:49.327 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): ceph-mgr-diskprediction-local-19.1.1-32 16 MB/s | 7.4 MB 00:00 2024-09-16T04:05:49.353 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 271 kB/s | 38 kB 00:00 2024-09-16T04:05:49.461 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-16T04:05:49.712 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:05:49.712 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T04:05:49.713 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T04:05:49.950 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T04:05:50.035 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T04:05:50.035 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:50.035 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:05:50.035 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T04:05:50.035 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:50.035 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:50.035 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:50.035 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T04:05:50.035 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T04:05:50.036 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T04:05:50.036 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T04:05:50.036 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T04:05:50.036 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T04:05:50.036 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T04:05:50.036 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T04:05:50.036 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:50.036 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:05:50.088 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T04:05:50.195 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-09-16T04:05:50.263 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T04:05:50.303 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-09-16T04:05:50.504 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.2 MB/s | 4.9 MB 00:01 2024-09-16T04:05:50.820 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-09-16T04:05:50.825 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:05:50.960 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:05:50.961 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-09-16T04:05:50.961 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:05:50.961 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-09-16T04:05:50.961 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:05:50.961 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T04:05:50.961 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:05:50.962 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-09-16T04:05:50.963 INFO:teuthology.orchestra.run.smithi174.stdout:Install 12 Packages 2024-09-16T04:05:50.963 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:05:50.963 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 38 M 2024-09-16T04:05:50.963 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 212 M 2024-09-16T04:05:50.963 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:05:51.365 INFO:teuthology.orchestra.run.smithi174.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 2.0 MB/s | 191 kB 00:00 2024-09-16T04:05:51.407 INFO:teuthology.orchestra.run.smithi174.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 5.8 MB/s | 801 kB 00:00 2024-09-16T04:05:51.445 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:05:51.516 INFO:teuthology.orchestra.run.smithi174.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 217 kB/s | 33 kB 00:00 2024-09-16T04:05:51.566 INFO:teuthology.orchestra.run.smithi174.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-16T04:05:51.616 INFO:teuthology.orchestra.run.smithi174.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-16T04:05:51.784 INFO:teuthology.orchestra.run.smithi174.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.9 MB/s | 3.0 MB 00:00 2024-09-16T04:05:51.842 INFO:teuthology.orchestra.run.smithi174.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-09-16T04:05:51.984 INFO:teuthology.orchestra.run.smithi174.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2024-09-16T04:05:52.067 INFO:teuthology.orchestra.run.smithi174.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-09-16T04:05:52.093 INFO:teuthology.orchestra.run.smithi174.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.3 MB/s | 250 kB 00:00 2024-09-16T04:05:52.517 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T04:05:52.544 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:05:52.551 INFO:teuthology.orchestra.run.smithi174.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-16T04:05:52.810 INFO:teuthology.orchestra.run.smithi174.stdout:(12/12): ceph-mgr-diskprediction-local-19.1.1-3 4.8 MB/s | 7.4 MB 00:01 2024-09-16T04:05:52.811 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:52.811 INFO:teuthology.orchestra.run.smithi174.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-16T04:05:53.176 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:05:53.202 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:05:53.202 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:05:53.658 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:05:53.659 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:05:53.870 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.6 MB/s | 16 MB 00:03 2024-09-16T04:05:53.871 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:53.872 INFO:teuthology.orchestra.run.smithi194.stdout:Total 7.1 MB/s | 38 MB 00:05 2024-09-16T04:05:54.231 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:05:54.257 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:05:54.257 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:05:54.438 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:05:54.533 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T04:05:54.584 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:05:54.689 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:05:54.689 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:05:54.690 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:05:54.727 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T04:05:54.912 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T04:05:54.950 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T04:05:55.026 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T04:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T04:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T04:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T04:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T04:05:55.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T04:05:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T04:05:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T04:05:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:05:55.198 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T04:05:55.460 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:05:55.532 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T04:05:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:05:55.665 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T04:05:55.728 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:55.728 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:05:55.728 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:55.728 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:55.729 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:05:55.730 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T04:05:55.730 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:05:55.730 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T04:05:55.797 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T04:05:55.913 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T04:05:55.943 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T04:05:56.025 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T04:05:56.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-rook 2024-09-16T04:05:56.630 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:18 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:05:56.680 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T04:05:56.769 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:05:56.770 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:56.770 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:05:56.770 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:56.770 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T04:05:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T04:05:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T04:05:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T04:05:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:56.772 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:05:56.772 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:05:56.772 INFO:teuthology.orchestra.run.smithi016.stdout:Install 13 Packages 2024-09-16T04:05:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:05:56.772 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 2.2 M 2024-09-16T04:05:56.773 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 26 M 2024-09-16T04:05:56.773 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:05:56.807 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T04:05:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:05:57.355 INFO:teuthology.orchestra.run.smithi016.stdout:(1/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 307 kB/s | 49 kB 00:00 2024-09-16T04:05:57.697 INFO:teuthology.orchestra.run.smithi016.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 38 kB/s | 19 kB 00:00 2024-09-16T04:05:57.789 INFO:teuthology.orchestra.run.smithi016.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 43 kB/s | 26 kB 00:00 2024-09-16T04:05:58.098 INFO:teuthology.orchestra.run.smithi016.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 299 kB/s | 222 kB 00:00 2024-09-16T04:05:58.099 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T04:05:58.124 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:05:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:05:58.440 INFO:teuthology.orchestra.run.smithi016.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 214 kB/s | 159 kB 00:00 2024-09-16T04:05:58.482 INFO:teuthology.orchestra.run.smithi016.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 778 kB/s | 32 kB 00:00 2024-09-16T04:05:58.507 INFO:teuthology.orchestra.run.smithi016.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 569 kB/s | 14 kB 00:00 2024-09-16T04:05:58.541 INFO:teuthology.orchestra.run.smithi016.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 6.5 MB/s | 220 kB 00:00 2024-09-16T04:05:58.625 INFO:teuthology.orchestra.run.smithi016.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 12 MB/s | 1.0 MB 00:00 2024-09-16T04:05:58.650 INFO:teuthology.orchestra.run.smithi016.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 324 kB/s | 279 kB 00:00 2024-09-16T04:05:58.675 INFO:teuthology.orchestra.run.smithi016.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-16T04:05:58.709 INFO:teuthology.orchestra.run.smithi016.stdout:(12/13): python3-requests-oauthlib-1.3.0-12.el9 88 kB/s | 54 kB 00:00 2024-09-16T04:05:58.742 INFO:teuthology.orchestra.run.smithi016.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 980 kB/s | 90 kB 00:00 2024-09-16T04:05:58.745 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:58.745 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.1 MB/s | 2.2 MB 00:01 2024-09-16T04:05:58.818 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:05:58.835 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:05:58.835 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:05:58.935 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:05:58.936 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:05:59.248 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T04:05:59.283 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:05:59.596 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:05:59.765 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T04:05:59.820 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T04:05:59.884 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T04:05:59.920 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T04:05:59.962 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T04:06:00.070 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T04:06:00.187 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T04:06:00.248 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T04:06:00.547 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T04:06:00.548 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:06:00.548 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T04:06:00.971 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T04:06:00.972 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T04:06:00.972 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T04:06:00.972 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T04:06:00.972 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T04:06:00.972 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:00.972 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:01.010 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T04:06:01.061 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T04:06:01.095 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T04:06:01.153 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T04:06:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:06:01.379 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-rook 2024-09-16T04:06:01.954 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:16 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:01.989 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:06:01.989 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T04:06:01.989 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:06:01.990 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:06:01.990 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T04:06:01.990 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T04:06:01.990 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T04:06:01.990 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T04:06:01.990 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T04:06:01.990 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T04:06:01.990 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:06:01.990 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T04:06:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:06:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T04:06:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T04:06:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T04:06:02.086 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:02.088 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:02.088 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:06:02.088 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:02.088 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:06:02.088 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T04:06:02.088 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-16T04:06:02.088 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:06:02.089 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:02.090 INFO:teuthology.orchestra.run.smithi174.stdout:Install 13 Packages 2024-09-16T04:06:02.090 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:02.090 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 2.2 M 2024-09-16T04:06:02.090 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 26 M 2024-09-16T04:06:02.090 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:06:02.363 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T04:06:02.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:02.363 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T04:06:02.364 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T04:06:02.365 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:02.365 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:02.413 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T04:06:02.413 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:02.413 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:02.413 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:02.413 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T04:06:02.413 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T04:06:02.413 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T04:06:02.413 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T04:06:02.413 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T04:06:02.414 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T04:06:02.414 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T04:06:02.414 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T04:06:02.414 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T04:06:02.414 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T04:06:02.414 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T04:06:02.414 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:02.414 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:02.523 INFO:teuthology.orchestra.run.smithi174.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 233 kB/s | 26 kB 00:00 2024-09-16T04:06:02.548 INFO:teuthology.orchestra.run.smithi174.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 142 kB/s | 19 kB 00:00 2024-09-16T04:06:02.573 INFO:teuthology.orchestra.run.smithi174.stdout:(3/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 307 kB/s | 49 kB 00:00 2024-09-16T04:06:02.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T04:06:02.623 INFO:teuthology.orchestra.run.smithi174.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-16T04:06:02.691 INFO:teuthology.orchestra.run.smithi174.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-16T04:06:02.716 INFO:teuthology.orchestra.run.smithi174.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 582 kB/s | 54 kB 00:00 2024-09-16T04:06:02.758 INFO:teuthology.orchestra.run.smithi174.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-16T04:06:02.783 INFO:teuthology.orchestra.run.smithi174.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-16T04:06:02.808 INFO:teuthology.orchestra.run.smithi174.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 166 kB/s | 14 kB 00:00 2024-09-16T04:06:02.842 INFO:teuthology.orchestra.run.smithi174.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-16T04:06:02.851 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-rook 2024-09-16T04:06:02.867 INFO:teuthology.orchestra.run.smithi174.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-16T04:06:02.892 INFO:teuthology.orchestra.run.smithi174.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T04:06:02.959 INFO:teuthology.orchestra.run.smithi174.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-09-16T04:06:02.962 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:02.962 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-16T04:06:03.029 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:06:03.043 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:06:03.043 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:06:03.133 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:06:03.135 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:06:03.215 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:03.357 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:03.357 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:03.357 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:06:03.357 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:03.358 INFO:teuthology.orchestra.run.smithi016.stdout:Install 6 Packages 2024-09-16T04:06:03.359 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:03.359 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 7.0 M 2024-09-16T04:06:03.359 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 33 M 2024-09-16T04:06:03.359 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:06:03.423 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:20 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 2.2 M 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 26 M 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:06:03.633 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:06:03.818 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T04:06:03.877 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi016.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T04:06:03.978 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T04:06:04.019 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T04:06:04.020 INFO:teuthology.orchestra.run.smithi016.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.5 MB/s | 548 kB 00:00 2024-09-16T04:06:04.054 INFO:teuthology.orchestra.run.smithi016.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-09-16T04:06:04.088 INFO:teuthology.orchestra.run.smithi016.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-16T04:06:04.121 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T04:06:04.174 INFO:teuthology.orchestra.run.smithi016.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-16T04:06:04.197 INFO:teuthology.orchestra.run.smithi016.stdout:(6/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 309 kB/s | 146 kB 00:00 2024-09-16T04:06:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:04.199 INFO:teuthology.orchestra.run.smithi016.stdout:Total 8.3 MB/s | 7.0 MB 00:00 2024-09-16T04:06:04.222 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T04:06:04.241 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 149 kB/s | 49 kB 00:00 2024-09-16T04:06:04.271 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:06:04.283 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:06:04.283 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:06:04.307 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T04:06:04.389 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:06:04.389 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:06:04.442 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 36 kB/s | 19 kB 00:00 2024-09-16T04:06:04.492 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 44 kB/s | 26 kB 00:00 2024-09-16T04:06:04.825 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:06:05.011 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T04:06:05.012 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T04:06:05.060 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T04:06:05.067 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T04:06:05.075 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 267 kB/s | 222 kB 00:00 2024-09-16T04:06:05.093 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T04:06:05.141 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T04:06:05.150 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 224 kB/s | 159 kB 00:00 2024-09-16T04:06:05.171 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:06:05.201 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 646 kB/s | 32 kB 00:00 2024-09-16T04:06:05.226 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 569 kB/s | 14 kB 00:00 2024-09-16T04:06:05.260 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 6.5 MB/s | 220 kB 00:00 2024-09-16T04:06:05.294 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 351 kB/s | 279 kB 00:00 2024-09-16T04:06:05.336 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-09-16T04:06:05.369 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-16T04:06:05.411 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-16T04:06:05.628 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T04:06:05.695 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 87 kB/s | 54 kB 00:00 2024-09-16T04:06:05.698 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:05.699 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.0 MB/s | 2.2 MB 00:02 2024-09-16T04:06:05.714 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T04:06:05.770 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:06:05.785 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T04:06:05.786 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:06:05.786 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:06:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:06:05.877 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:06:05.877 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:06:05.964 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:06:05.964 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T04:06:05.964 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T04:06:05.964 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T04:06:05.964 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T04:06:05.964 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T04:06:05.964 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T04:06:05.964 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T04:06:05.965 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T04:06:05.965 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T04:06:05.965 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T04:06:05.965 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T04:06:05.965 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T04:06:06.325 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T04:06:06.325 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:06.325 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:06.326 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:06.382 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:06:06.549 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T04:06:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:06:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T04:06:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T04:06:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T04:06:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T04:06:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T04:06:06.599 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T04:06:06.604 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T04:06:06.668 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T04:06:06.704 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T04:06:06.746 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T04:06:06.850 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T04:06:06.862 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:06.863 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:06.965 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T04:06:07.023 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T04:06:07.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-fuse 2024-09-16T04:06:07.168 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:22 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:07.304 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T04:06:07.305 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T04:06:07.306 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T04:06:07.306 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:07.306 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:06:07.306 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:07.306 INFO:teuthology.orchestra.run.smithi174.stdout:Install 6 Packages 2024-09-16T04:06:07.306 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:07.306 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 7.0 M 2024-09-16T04:06:07.307 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 33 M 2024-09-16T04:06:07.307 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:06:07.704 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:07.731 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T04:06:07.778 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T04:06:07.812 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T04:06:07.844 INFO:teuthology.orchestra.run.smithi174.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-16T04:06:07.845 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:07.846 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:07.846 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:06:07.846 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:07.846 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:06:07.846 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T04:06:07.847 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-16T04:06:07.847 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T04:06:07.847 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:07.847 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:06:07.847 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:07.847 INFO:teuthology.orchestra.run.smithi016.stdout:Install 2 Packages 2024-09-16T04:06:07.847 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:07.848 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 976 k 2024-09-16T04:06:07.848 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.9 M 2024-09-16T04:06:07.848 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:06:07.859 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T04:06:07.882 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:06:07.886 INFO:teuthology.orchestra.run.smithi174.stdout:(2/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 548 kB/s | 146 kB 00:00 2024-09-16T04:06:07.928 INFO:teuthology.orchestra.run.smithi174.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-16T04:06:07.953 INFO:teuthology.orchestra.run.smithi174.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-16T04:06:07.970 INFO:teuthology.orchestra.run.smithi174.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 690 kB/s | 58 kB 00:00 2024-09-16T04:06:08.075 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 638 kB/s | 80 kB 00:00 2024-09-16T04:06:08.217 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.3 MB/s | 896 kB 00:00 2024-09-16T04:06:08.217 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:08.218 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.6 MB/s | 976 kB 00:00 2024-09-16T04:06:08.230 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:06:08.253 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:06:08.253 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:06:08.340 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:06:08.340 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:06:08.542 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T04:06:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T04:06:08.650 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T04:06:08.650 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T04:06:08.691 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T04:06:08.762 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:06:08.991 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T04:06:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T04:06:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T04:06:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T04:06:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T04:06:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T04:06:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:08.993 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:09.029 INFO:teuthology.orchestra.run.smithi174.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-09-16T04:06:09.032 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:09.032 INFO:teuthology.orchestra.run.smithi174.stdout:Total 4.1 MB/s | 7.0 MB 00:01 2024-09-16T04:06:09.107 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:06:09.119 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:06:09.119 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:06:09.219 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:06:09.221 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:06:09.287 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T04:06:09.396 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:06:09.396 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T04:06:09.530 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:06:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T04:06:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:09.655 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:06:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T04:06:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:09.655 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:09.700 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T04:06:09.747 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T04:06:09.861 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:09.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-volume 2024-09-16T04:06:09.993 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:09.994 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:09.994 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:06:09.994 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:09.994 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:06:09.994 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T04:06:09.994 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-16T04:06:09.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T04:06:09.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T04:06:09.995 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T04:06:09.995 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T04:06:09.995 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T04:06:09.995 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:09.995 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:06:09.995 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:09.995 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-09-16T04:06:09.995 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:09.996 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 7.0 M 2024-09-16T04:06:09.996 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 33 M 2024-09-16T04:06:09.996 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:06:10.263 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T04:06:10.350 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T04:06:10.412 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T04:06:10.443 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:06:10.552 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:10.654 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-16T04:06:10.679 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 696 kB/s | 146 kB 00:00 2024-09-16T04:06:10.693 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:10.694 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:10.695 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-16T04:06:10.695 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:10.695 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:06:10.695 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T04:06:10.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:10.695 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:06:10.696 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:10.696 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-16T04:06:10.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:10.696 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 285 k 2024-09-16T04:06:10.696 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 1.3 M 2024-09-16T04:06:10.697 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:06:10.729 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-16T04:06:10.755 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 766 kB/s | 58 kB 00:00 2024-09-16T04:06:10.780 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-16T04:06:10.906 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.3 MB/s | 285 kB 00:00 2024-09-16T04:06:10.907 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:10.907 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-16T04:06:10.907 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:06:10.916 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:06:10.916 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:06:10.948 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:06:10.949 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:06:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:06:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:06:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T04:06:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T04:06:11.224 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T04:06:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T04:06:11.225 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T04:06:11.258 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:06:11.275 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:06:11.275 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:06:11.275 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T04:06:11.275 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T04:06:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:11.515 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:11.515 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T04:06:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T04:06:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T04:06:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T04:06:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T04:06:11.516 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:11.516 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:11.769 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-fuse 2024-09-16T04:06:11.813 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.4 MB/s | 6.0 MB 00:01 2024-09-16T04:06:11.816 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:11.816 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.9 MB/s | 7.0 MB 00:01 2024-09-16T04:06:11.887 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:06:11.898 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:06:11.898 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:06:11.997 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:06:11.997 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:06:12.302 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:06:12.303 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:06:12.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:12.303 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:06:12.303 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:12.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:12.303 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:12.338 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:27 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:12.470 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T04:06:12.472 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:06:12.473 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:12.474 INFO:teuthology.orchestra.run.smithi174.stdout:Install 2 Packages 2024-09-16T04:06:12.474 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:12.474 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 976 k 2024-09-16T04:06:12.474 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 2.9 M 2024-09-16T04:06:12.474 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:06:12.523 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T04:06:12.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados-devel 2024-09-16T04:06:12.647 INFO:teuthology.orchestra.run.smithi174.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 527 kB/s | 80 kB 00:00 2024-09-16T04:06:12.789 INFO:teuthology.orchestra.run.smithi174.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.0 MB/s | 896 kB 00:00 2024-09-16T04:06:12.789 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:12.790 INFO:teuthology.orchestra.run.smithi174.stdout:Total 3.0 MB/s | 976 kB 00:00 2024-09-16T04:06:12.801 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:06:12.823 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:06:12.824 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:06:12.905 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:06:12.906 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:06:13.040 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T04:06:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:06:13.134 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T04:06:13.172 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T04:06:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:06:13.233 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T04:06:13.304 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:06:13.310 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-16T04:06:13.311 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:13.312 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 127 k 2024-09-16T04:06:13.312 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 456 k 2024-09-16T04:06:13.312 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:06:13.520 INFO:teuthology.orchestra.run.smithi016.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 611 kB/s | 127 kB 00:00 2024-09-16T04:06:13.520 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:13.521 INFO:teuthology.orchestra.run.smithi016.stdout:Total 606 kB/s | 127 kB 00:00 2024-09-16T04:06:13.521 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:06:13.527 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:06:13.527 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:06:13.600 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:06:13.601 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:06:13.771 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:06:13.880 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:13.952 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:06:13.952 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T04:06:14.008 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:06:14.008 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T04:06:14.008 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T04:06:14.008 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T04:06:14.008 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T04:06:14.009 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T04:06:14.273 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T04:06:14.273 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:14.274 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:14.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:14.274 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T04:06:14.274 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T04:06:14.274 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T04:06:14.274 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T04:06:14.274 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T04:06:14.275 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:14.275 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:14.275 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T04:06:14.276 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:14.276 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:14.276 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T04:06:14.276 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:14.276 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:14.345 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:14.536 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-09-16T04:06:14.541 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-volume 2024-09-16T04:06:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:14.607 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:06:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:14.607 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:14.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs2 2024-09-16T04:06:15.105 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:15.126 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:15.238 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:15.239 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-09-16T04:06:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:15.240 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 976 k 2024-09-16T04:06:15.240 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.9 M 2024-09-16T04:06:15.240 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:06:15.268 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:15.268 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:15.269 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-16T04:06:15.269 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:15.269 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:06:15.269 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T04:06:15.269 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:15.269 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:06:15.269 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:15.269 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-16T04:06:15.269 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:15.270 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 285 k 2024-09-16T04:06:15.270 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 1.3 M 2024-09-16T04:06:15.270 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:06:15.381 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 668 kB/s | 80 kB 00:00 2024-09-16T04:06:15.438 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:15.510 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-16T04:06:15.511 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:15.511 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T04:06:15.511 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:06:15.519 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:06:15.519 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:06:15.533 INFO:teuthology.orchestra.run.smithi016.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:15.548 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.1 MB/s | 896 kB 00:00 2024-09-16T04:06:15.548 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:15.549 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.1 MB/s | 976 kB 00:00 2024-09-16T04:06:15.556 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:06:15.556 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:06:15.561 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:06:15.579 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:15.580 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-16T04:06:15.581 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:15.582 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:06:15.583 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:06:15.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs-devel 2024-09-16T04:06:15.664 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:06:15.664 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:06:15.771 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:06:15.837 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:06:15.848 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:06:15.853 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:06:15.853 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:06:15.853 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T04:06:15.853 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:16.000 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T04:06:16.072 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:06:16.243 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:16.382 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:16.382 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-09-16T04:06:16.382 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:16.382 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:06:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T04:06:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:16.382 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:06:16.383 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:16.383 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-16T04:06:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:16.383 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 32 k 2024-09-16T04:06:16.383 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 142 k 2024-09-16T04:06:16.383 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:06:16.533 INFO:teuthology.orchestra.run.smithi016.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 212 kB/s | 32 kB 00:00 2024-09-16T04:06:16.533 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:16.534 INFO:teuthology.orchestra.run.smithi016.stdout:Total 210 kB/s | 32 kB 00:00 2024-09-16T04:06:16.534 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:06:16.540 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:06:16.540 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:06:16.559 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:06:16.560 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:06:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:06:16.766 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:06:16.766 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T04:06:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:16.887 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:06:16.887 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:16.887 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:16.887 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:16.887 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:16.887 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:17.042 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T04:06:17.042 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:17.042 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:17.042 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T04:06:17.042 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:17.042 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:17.184 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados-devel 2024-09-16T04:06:17.327 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-volume 2024-09-16T04:06:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:17.385 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:06:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:17.386 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:17.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados2 2024-09-16T04:06:17.762 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:17.901 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:17.901 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:17.901 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 127 k 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 456 k 2024-09-16T04:06:17.902 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:06:17.907 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:18.043 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:18.043 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-16T04:06:18.043 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:18.043 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:06:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T04:06:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:18.044 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:06:18.044 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:18.044 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-16T04:06:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:18.045 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 285 k 2024-09-16T04:06:18.045 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.3 M 2024-09-16T04:06:18.045 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:06:18.098 INFO:teuthology.orchestra.run.smithi174.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 649 kB/s | 127 kB 00:00 2024-09-16T04:06:18.098 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:18.099 INFO:teuthology.orchestra.run.smithi174.stdout:Total 644 kB/s | 127 kB 00:00 2024-09-16T04:06:18.099 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:06:18.104 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:06:18.104 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:06:18.168 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:18.176 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:06:18.176 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:06:18.264 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:18.306 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.1 MB/s | 285 kB 00:00 2024-09-16T04:06:18.307 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:18.307 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-16T04:06:18.307 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:06:18.309 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:18.311 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-16T04:06:18.311 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:18.315 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:06:18.315 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:06:18.338 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:06:18.347 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:06:18.348 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:06:18.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librbd1 2024-09-16T04:06:18.437 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:18.563 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:06:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:06:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:06:18.653 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:06:18.654 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T04:06:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:18.878 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:18.978 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:19.073 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:19.119 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:19.120 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-16T04:06:19.120 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:19.131 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:19.131 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:19.131 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:19.131 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:19.131 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:19.131 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:19.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rados 2024-09-16T04:06:19.375 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs2 2024-09-16T04:06:19.674 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:06:19.674 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:19.674 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:19.674 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:06:19.674 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:19.675 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:19.786 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:19.881 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:19.927 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:19.928 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-16T04:06:19.928 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:19.959 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:19.983 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-09-16T04:06:20.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rgw 2024-09-16T04:06:20.054 INFO:teuthology.orchestra.run.smithi174.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:20.098 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:20.100 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-16T04:06:20.100 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:20.192 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs-devel 2024-09-16T04:06:20.569 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:20.591 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:20.685 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:20.703 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:20.704 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:20.704 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 127 k 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 456 k 2024-09-16T04:06:20.705 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:06:20.730 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:20.731 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-16T04:06:20.731 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:20.758 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:20.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-cephfs 2024-09-16T04:06:20.891 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:20.892 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:20.892 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-09-16T04:06:20.892 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:20.892 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:06:20.893 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T04:06:20.893 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:20.893 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:06:20.893 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:20.893 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-16T04:06:20.893 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:20.893 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 32 k 2024-09-16T04:06:20.893 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 142 k 2024-09-16T04:06:20.893 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:06:20.907 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 629 kB/s | 127 kB 00:00 2024-09-16T04:06:20.908 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:20.908 INFO:teuthology.orchestra.run.smithi194.stdout:Total 622 kB/s | 127 kB 00:00 2024-09-16T04:06:20.908 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:06:20.914 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:06:20.914 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:06:20.982 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:06:20.983 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:06:21.042 INFO:teuthology.orchestra.run.smithi174.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 212 kB/s | 32 kB 00:00 2024-09-16T04:06:21.043 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:21.043 INFO:teuthology.orchestra.run.smithi174.stdout:Total 210 kB/s | 32 kB 00:00 2024-09-16T04:06:21.043 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:06:21.047 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:06:21.047 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:06:21.065 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:06:21.065 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:06:21.144 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:06:21.151 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:06:21.246 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:21.248 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:21.387 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:21.482 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:21.528 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:21.529 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-16T04:06:21.529 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:21.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rbd 2024-09-16T04:06:21.619 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:21.701 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:21.871 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:21.871 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:21.872 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:21.872 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:21.872 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:21.872 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:21.963 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:21.963 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:21.963 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:21.963 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:21.963 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:22.048 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados2 2024-09-16T04:06:22.198 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:22.207 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-09-16T04:06:22.293 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:22.339 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:22.340 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-16T04:06:22.341 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:22.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-fuse 2024-09-16T04:06:22.618 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:22.709 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:22.752 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:22.753 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-16T04:06:22.754 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:22.787 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:22.819 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librbd1 2024-09-16T04:06:22.880 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:22.926 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:22.927 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-16T04:06:22.927 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:23.010 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-09-16T04:06:23.013 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:23.155 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:23.156 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:23.156 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-09-16T04:06:23.156 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:23.157 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:06:23.157 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T04:06:23.157 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:23.157 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:06:23.158 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:23.158 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-16T04:06:23.158 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:23.158 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 85 k 2024-09-16T04:06:23.158 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 231 k 2024-09-16T04:06:23.158 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:06:23.345 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 454 kB/s | 85 kB 00:00 2024-09-16T04:06:23.346 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:23.346 INFO:teuthology.orchestra.run.smithi016.stdout:Total 450 kB/s | 85 kB 00:00 2024-09-16T04:06:23.346 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:06:23.353 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:06:23.354 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:06:23.391 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:23.421 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:06:23.421 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:06:23.487 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:23.533 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:23.534 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-16T04:06:23.535 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:23.574 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:06:23.575 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:23.616 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rados 2024-09-16T04:06:23.687 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:23.710 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:23.710 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:23.710 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-09-16T04:06:23.710 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:23.710 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:06:23.710 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T04:06:23.711 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:23.711 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:06:23.711 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:23.711 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-16T04:06:23.711 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:23.711 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 32 k 2024-09-16T04:06:23.711 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 142 k 2024-09-16T04:06:23.711 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:06:23.851 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 227 kB/s | 32 kB 00:00 2024-09-16T04:06:23.851 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:23.852 INFO:teuthology.orchestra.run.smithi194.stdout:Total 225 kB/s | 32 kB 00:00 2024-09-16T04:06:23.852 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:06:23.855 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:06:23.855 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:06:23.875 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:06:23.876 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:06:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:24.218 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:24.313 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:24.354 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:06:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:24.355 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:24.358 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:24.359 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-16T04:06:24.359 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:24.422 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rgw 2024-09-16T04:06:24.456 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:24.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-mirror 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:24.915 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-09-16T04:06:24.992 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:25.083 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:25.125 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:25.126 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-16T04:06:25.126 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:25.188 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-cephfs 2024-09-16T04:06:25.233 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:25.375 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-09-16T04:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T04:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-16T04:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 3.1 M 2024-09-16T04:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 13 M 2024-09-16T04:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:06:25.488 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:25.580 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:25.623 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:25.624 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-16T04:06:25.624 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:25.709 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-09-16T04:06:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 8.9 MB/s | 3.1 MB 00:00 2024-09-16T04:06:25.725 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:Total 8.9 MB/s | 3.1 MB 00:00 2024-09-16T04:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:06:25.754 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:06:25.821 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:06:25.845 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:25.888 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:25.889 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-16T04:06:25.889 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:25.955 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rbd 2024-09-16T04:06:26.276 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:06:26.359 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:26.367 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:26.396 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:26.396 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:06:26.397 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T04:06:26.397 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:06:26.397 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:06:26.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:26.410 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:26.412 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-16T04:06:26.412 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:26.502 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados 2024-09-16T04:06:26.523 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:26.613 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:26.656 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:26.657 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-16T04:06:26.657 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:26.721 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-fuse 2024-09-16T04:06:27.068 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:27.159 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:27.202 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:27.203 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-16T04:06:27.203 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:27.284 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw 2024-09-16T04:06:27.287 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:27.422 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:27.423 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 85 k 2024-09-16T04:06:27.424 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 231 k 2024-09-16T04:06:27.424 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:06:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:27.569 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:06:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:27.569 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:27.569 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:27.629 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 415 kB/s | 85 kB 00:00 2024-09-16T04:06:27.630 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:27.630 INFO:teuthology.orchestra.run.smithi174.stdout:Total 412 kB/s | 85 kB 00:00 2024-09-16T04:06:27.630 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:06:27.637 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:06:27.637 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:06:27.700 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:06:27.701 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:06:27.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-nbd 2024-09-16T04:06:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:06:27.853 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:27.945 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:27.951 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:27.988 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:27.989 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-16T04:06:27.989 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:28.067 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs 2024-09-16T04:06:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:28.445 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 04:04:38 AM UTC. 2024-09-16T04:06:28.587 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-16T04:06:28.587 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:28.587 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-09-16T04:06:28.587 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:28.587 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-16T04:06:28.587 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T04:06:28.587 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:28.587 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-16T04:06:28.588 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-16T04:06:28.588 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-16T04:06:28.588 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:28.588 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 170 k 2024-09-16T04:06:28.588 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 494 k 2024-09-16T04:06:28.588 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-16T04:06:28.637 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:28.655 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:28.655 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:28.655 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:28.656 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:28.656 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:28.656 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:28.728 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:28.771 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:28.772 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-16T04:06:28.772 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:28.797 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 819 kB/s | 170 kB 00:00 2024-09-16T04:06:28.797 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:28.797 INFO:teuthology.orchestra.run.smithi016.stdout:Total 813 kB/s | 170 kB 00:00 2024-09-16T04:06:28.798 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-16T04:06:28.806 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-16T04:06:28.806 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-16T04:06:28.865 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd 2024-09-16T04:06:28.874 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-16T04:06:28.875 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-16T04:06:28.913 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-mirror 2024-09-16T04:06:29.143 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-16T04:06:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:29.449 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:29.497 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:29.546 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:06:29.592 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:29.593 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-16T04:06:29.593 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:29.637 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:29.638 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:29.638 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-09-16T04:06:29.638 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:29.638 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:06:29.638 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T04:06:29.640 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:29.640 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:06:29.640 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:29.640 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-16T04:06:29.640 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:29.641 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 3.1 M 2024-09-16T04:06:29.641 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 13 M 2024-09-16T04:06:29.641 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:06:29.674 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:29.680 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-09-16T04:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-16T04:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-16T04:06:29.982 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-09-16T04:06:29.983 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:29.983 INFO:teuthology.orchestra.run.smithi174.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-09-16T04:06:29.983 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:06:29.992 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:06:29.992 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:06:30.073 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:06:30.074 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:06:30.202 DEBUG:teuthology.parallel:result is None 2024-09-16T04:06:30.272 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:30.412 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:30.412 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 85 k 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 231 k 2024-09-16T04:06:30.413 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:06:30.529 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:06:30.591 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:30.595 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 468 kB/s | 85 kB 00:00 2024-09-16T04:06:30.596 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:30.597 INFO:teuthology.orchestra.run.smithi194.stdout:Total 463 kB/s | 85 kB 00:00 2024-09-16T04:06:30.597 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:06:30.604 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:06:30.604 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:06:30.617 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:30.617 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:06:30.618 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T04:06:30.618 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:06:30.618 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:06:30.618 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:30.668 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:06:30.669 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:06:30.812 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:06:30.918 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:31.326 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:31.579 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:31.579 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:31.579 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:31.579 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:31.579 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:31.579 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:31.748 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:31.748 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:31.748 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:31.748 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:31.748 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:31.748 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:31.874 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-09-16T04:06:31.987 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-nbd 2024-09-16T04:06:32.457 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:32.572 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 04:04:45 AM UTC. 2024-09-16T04:06:32.598 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:32.599 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:32.599 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-09-16T04:06:32.599 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:32.599 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:06:32.599 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T04:06:32.599 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:32.599 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:06:32.599 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:32.599 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-16T04:06:32.600 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:32.600 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.1 M 2024-09-16T04:06:32.600 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 13 M 2024-09-16T04:06:32.600 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:06:32.707 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-16T04:06:32.707 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 170 k 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 494 k 2024-09-16T04:06:32.708 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-16T04:06:32.932 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 9.4 MB/s | 3.1 MB 00:00 2024-09-16T04:06:32.932 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:32.933 INFO:teuthology.orchestra.run.smithi194.stdout:Total 9.3 MB/s | 3.1 MB 00:00 2024-09-16T04:06:32.934 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:06:32.936 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 753 kB/s | 170 kB 00:00 2024-09-16T04:06:32.936 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:32.936 INFO:teuthology.orchestra.run.smithi174.stdout:Total 748 kB/s | 170 kB 00:00 2024-09-16T04:06:32.936 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-16T04:06:32.941 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:06:32.941 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:06:32.943 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-16T04:06:32.943 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-16T04:06:33.009 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-16T04:06:33.009 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-16T04:06:33.022 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:06:33.022 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:06:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-16T04:06:33.291 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:33.451 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:06:33.516 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:33.542 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:06:33.542 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T04:06:33.542 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:06:33.542 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:06:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:33.727 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:33.984 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:33.984 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:33.984 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-16T04:06:33.985 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:33.985 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:33.985 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-16T04:06:34.246 DEBUG:teuthology.parallel:result is None 2024-09-16T04:06:34.632 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:34.632 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:34.632 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:34.633 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:34.633 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:34.633 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:34.906 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-09-16T04:06:35.493 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Sep 2024 04:04:43 AM UTC. 2024-09-16T04:06:35.629 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-16T04:06:35.629 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:35.629 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 170 k 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 494 k 2024-09-16T04:06:35.630 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-16T04:06:35.859 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 744 kB/s | 170 kB 00:00 2024-09-16T04:06:35.860 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:06:35.860 INFO:teuthology.orchestra.run.smithi194.stdout:Total 740 kB/s | 170 kB 00:00 2024-09-16T04:06:35.860 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-16T04:06:35.868 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-16T04:06:35.868 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-16T04:06:35.932 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-16T04:06:35.932 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-16T04:06:36.097 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-16T04:06:36.198 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:36.622 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:36.900 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:06:36.900 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:36.900 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-16T04:06:36.900 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:06:36.900 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:36.900 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-16T04:06:37.181 DEBUG:teuthology.parallel:result is None 2024-09-16T04:06:37.182 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:06:37.326 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T04:06:37.352 INFO:teuthology.orchestra.run.smithi016.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T04:06:37.353 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T04:06:37.353 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T04:06:37.355 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:06:37.502 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T04:06:37.527 INFO:teuthology.orchestra.run.smithi174.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T04:06:37.528 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T04:06:37.528 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T04:06:37.530 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:06:37.664 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T04:06:37.689 INFO:teuthology.orchestra.run.smithi194.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T04:06:37.689 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T04:06:37.689 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T04:06:37.691 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T04:06:37.692 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:37.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:06:37.724 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:06:37.724 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:06:37.755 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:06:37.755 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:06:37.794 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T04:06:37.794 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:37.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:06:37.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:06:37.894 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:06:37.894 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:06:37.924 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:06:37.993 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:06:37.994 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:06:38.021 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:06:38.091 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T04:06:38.091 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:38.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:06:38.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:06:38.193 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:06:38.193 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:06:38.224 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:06:38.293 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:06:38.293 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:06:38.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:06:38.389 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T04:06:38.390 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:38.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:06:38.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:06:38.492 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:06:38.492 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:06:38.522 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:06:38.591 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:06:38.591 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:06:38.619 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:06:38.688 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T04:06:38.767 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T04:06:38.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2024-09-16T04:06:38.770 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /var/log/ceph 2024-09-16T04:06:38.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /var/log/ceph 2024-09-16T04:06:38.799 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T04:06:38.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:06:38.840 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:06:38.843 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:06:38.870 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T04:06:38.870 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:06:38.910 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:06:38.913 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:06:38.949 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T04:06:38.949 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T04:06:38.950 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966', 'branch': 'squid', 'description': 'rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905966', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'], ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'], ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNVGk+mTN4O/+B5mgT/ywEsv0T8BJL55lyldEerkLi1YKLJUTMZ/DoelrQRNVFsliLqbu7djZNp/ZU2PzVVTU0o=', 'smithi174.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPldhjKpiowGAQ3cIOs6Lcq2K5LV+nejpkxm7Dz2SS/Dalpd6p7+7typ5Eajd7xIV33sgRL3a4IwJQfS+1PCgjA=', 'smithi194.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvw05pPvP+wC8ydK4DsWixLJPlEJTGPSP9ZIXCNc1Josv6Kd4FwyCc2T1tQlQbBEGpwQDdNzwhINTGU+vyKrIs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 4, 'l': 3, 'm': 2, 'name': 'lrcprofile', 'plugin': 'lrc'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T04:06:38.950 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:06:38.980 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:06:38.983 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:06:38.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:06:39.038 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:06:39.041 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:06:39.067 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:39.067 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:06:39.123 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:06:39.123 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-09-16T04:06:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:06:39.181 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:39.181 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T04:06:39.181 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:39.181 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:39.182 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-16 04:06:26.651844286 +0000 2024-09-16T04:06:39.182 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-16 04:04:01.586093757 +0000 2024-09-16T04:06:39.182 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-16 04:04:01.586093757 +0000 2024-09-16T04:06:39.182 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-16 04:04:01.586093757 +0000 2024-09-16T04:06:39.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:06:39.248 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-16T04:06:39.249 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-16T04:06:39.249 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000102789 s, 5.0 MB/s 2024-09-16T04:06:39.250 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:06:39.308 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-09-16T04:06:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:06:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:39.366 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-16T04:06:39.366 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:39.366 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:39.366 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-16 04:06:26.651844286 +0000 2024-09-16T04:06:39.366 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-16 04:04:01.913088905 +0000 2024-09-16T04:06:39.366 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-16 04:04:01.913088905 +0000 2024-09-16T04:06:39.366 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-16 04:04:01.913088905 +0000 2024-09-16T04:06:39.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:06:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-16T04:06:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-16T04:06:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000114835 s, 4.5 MB/s 2024-09-16T04:06:39.433 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:06:39.490 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-09-16T04:06:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:06:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:39.548 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-16T04:06:39.548 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:39.548 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:39.548 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-16 04:06:26.652844271 +0000 2024-09-16T04:06:39.548 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-16 04:04:02.227084245 +0000 2024-09-16T04:06:39.548 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-16 04:04:02.227084245 +0000 2024-09-16T04:06:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-16 04:04:02.227084245 +0000 2024-09-16T04:06:39.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:06:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-16T04:06:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-16T04:06:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000111867 s, 4.6 MB/s 2024-09-16T04:06:39.614 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:06:39.672 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-09-16T04:06:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:06:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:39.730 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-16T04:06:39.730 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:39.730 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:39.730 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-16 04:06:26.652844271 +0000 2024-09-16T04:06:39.730 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-16 04:04:02.594078798 +0000 2024-09-16T04:06:39.730 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-16 04:04:02.594078798 +0000 2024-09-16T04:06:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-16 04:04:02.594078798 +0000 2024-09-16T04:06:39.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:06:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-16T04:06:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-16T04:06:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000100451 s, 5.1 MB/s 2024-09-16T04:06:39.796 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:06:39.854 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:39.854 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:06:39.854 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:06:39.873 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:06:39.874 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-09-16T04:06:39.934 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:06:39.934 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:39.934 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T04:06:39.934 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:39.934 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:39.934 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-16 04:06:30.886202220 +0000 2024-09-16T04:06:39.934 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-16 04:04:00.924006347 +0000 2024-09-16T04:06:39.934 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-16 04:04:00.924006347 +0000 2024-09-16T04:06:39.934 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-16 04:04:00.924006347 +0000 2024-09-16T04:06:39.934 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:06:40.001 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-16T04:06:40.001 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-16T04:06:40.001 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000276908 s, 1.8 MB/s 2024-09-16T04:06:40.002 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:06:40.061 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-09-16T04:06:40.121 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:06:40.121 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:40.121 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T04:06:40.121 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:40.121 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:40.121 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-16 04:06:30.886202220 +0000 2024-09-16T04:06:40.121 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-16 04:04:01.255002780 +0000 2024-09-16T04:06:40.121 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-16 04:04:01.255002780 +0000 2024-09-16T04:06:40.121 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-16 04:04:01.255002780 +0000 2024-09-16T04:06:40.121 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:06:40.186 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-16T04:06:40.186 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-16T04:06:40.186 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000205765 s, 2.5 MB/s 2024-09-16T04:06:40.187 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:06:40.245 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-09-16T04:06:40.302 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:06:40.302 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:40.302 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-16T04:06:40.302 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:40.302 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:40.302 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-16 04:06:30.886202220 +0000 2024-09-16T04:06:40.302 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-16 04:04:01.582999244 +0000 2024-09-16T04:06:40.302 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-16 04:04:01.582999244 +0000 2024-09-16T04:06:40.302 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-16 04:04:01.582999244 +0000 2024-09-16T04:06:40.302 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:06:40.366 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-16T04:06:40.366 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-16T04:06:40.366 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000228949 s, 2.2 MB/s 2024-09-16T04:06:40.367 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:06:40.424 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-09-16T04:06:40.480 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:06:40.480 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:40.480 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T04:06:40.480 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:40.480 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:40.480 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-16 04:06:30.887202208 +0000 2024-09-16T04:06:40.480 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-16 04:04:01.896995860 +0000 2024-09-16T04:06:40.480 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-16 04:04:01.896995860 +0000 2024-09-16T04:06:40.480 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-16 04:04:01.896995860 +0000 2024-09-16T04:06:40.481 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:06:40.544 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-16T04:06:40.544 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-16T04:06:40.544 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000214621 s, 2.4 MB/s 2024-09-16T04:06:40.545 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:06:40.602 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:40.602 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:06:40.602 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:06:40.620 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:06:40.620 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-09-16T04:06:40.677 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:06:40.677 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T04:06:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-16 04:06:33.794475497 +0000 2024-09-16T04:06:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-16 04:03:59.515151741 +0000 2024-09-16T04:06:40.678 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-16 04:03:59.515151741 +0000 2024-09-16T04:06:40.678 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-16 04:03:59.515151741 +0000 2024-09-16T04:06:40.678 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:06:40.743 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-16T04:06:40.744 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-16T04:06:40.744 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000204236 s, 2.5 MB/s 2024-09-16T04:06:40.745 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:06:40.803 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-09-16T04:06:40.860 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:06:40.860 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:40.860 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T04:06:40.860 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:40.860 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:40.860 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-16 04:06:33.794475497 +0000 2024-09-16T04:06:40.860 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-16 04:03:59.851144845 +0000 2024-09-16T04:06:40.860 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-16 04:03:59.851144845 +0000 2024-09-16T04:06:40.860 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-16 04:03:59.851144845 +0000 2024-09-16T04:06:40.861 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:06:40.924 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-16T04:06:40.924 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-16T04:06:40.924 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000245807 s, 2.1 MB/s 2024-09-16T04:06:40.925 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:06:40.982 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-09-16T04:06:41.037 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:06:41.037 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:41.037 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T04:06:41.037 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:41.038 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:41.038 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-16 04:06:33.794475497 +0000 2024-09-16T04:06:41.038 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-16 04:04:00.173138236 +0000 2024-09-16T04:06:41.038 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-16 04:04:00.173138236 +0000 2024-09-16T04:06:41.038 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-16 04:04:00.173138236 +0000 2024-09-16T04:06:41.038 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:06:41.101 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-16T04:06:41.101 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-16T04:06:41.101 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.00018907 s, 2.7 MB/s 2024-09-16T04:06:41.102 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:06:41.159 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-09-16T04:06:41.214 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:06:41.214 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:06:41.214 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T04:06:41.214 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:06:41.214 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:06:41.214 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-16 04:06:33.795475476 +0000 2024-09-16T04:06:41.214 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-16 04:04:00.496131608 +0000 2024-09-16T04:06:41.214 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-16 04:04:00.496131608 +0000 2024-09-16T04:06:41.214 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-16 04:04:00.496131608 +0000 2024-09-16T04:06:41.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:06:41.276 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-16T04:06:41.277 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-16T04:06:41.277 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000208413 s, 2.5 MB/s 2024-09-16T04:06:41.278 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:06:41.335 INFO:tasks.ceph:osd dev map: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:41.335 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'}} 2024-09-16T04:06:41.335 INFO:tasks.ceph:Generating config... 2024-09-16T04:06:41.336 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T04:06:41.336 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-16T04:06:41.336 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-16T04:06:41.336 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2024-09-16T04:06:41.337 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd max backfills = 6 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-16T04:06:41.338 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-16T04:06:41.338 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T04:06:41.338 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T04:06:41.338 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-16T04:06:41.338 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-16T04:06:41.339 INFO:tasks.ceph:[global] ms inject delay type = osd 2024-09-16T04:06:41.339 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-16T04:06:41.339 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-16T04:06:41.339 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-16T04:06:41.339 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T04:06:41.339 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T04:06:41.339 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T04:06:41.339 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T04:06:41.339 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T04:06:41.339 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T04:06:41.339 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T04:06:41.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T04:06:41.404 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T04:06:41.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T04:06:41.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:06:41.482 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.16'), ('mon.b', '172.21.15.174'), ('mon.c', '172.21.15.194')] 2024-09-16T04:06:41.482 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.16,172.21.15.174,172.21.15.194', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-16T04:06:41.483 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:41.483 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:06:41.538 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.16 --add b 172.21.15.174 --add c 172.21.15.194 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:41.627 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid 44381953-4509-4b15-853e-3887928dfa29 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 44381953-4509-4b15-853e-3887928dfa29 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2024-09-16T04:06:41.626540+0000 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-16T04:06:41.626540+0000 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 17 (quincy) 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:election_strategy: 3 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:2: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-09-16T04:06:41.628 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T04:06:41.630 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:06:41.646 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 44381953-4509-4b15-853e-3887928dfa29... 2024-09-16T04:06:41.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:06:41.688 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:06:41.691 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:06:41.730 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-09-16T04:06:41.730 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-09-16T04:06:41.730 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.731 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-09-16T04:06:41.732 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:06:41.733 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.174,172.21.15.194" 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: mon client directed command retry = 5 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: mon election default strategy = 3 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject delay max = 1 2024-09-16T04:06:41.734 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject delay probability = 0.005 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject delay type = osd 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject internal delays = 0.002 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject socket failures = 2500 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 44381953-4509-4b15-853e-3887928dfa29 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-09-16T04:06:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 120 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.736 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-09-16T04:06:41.737 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: debug monc = 20 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: bluefs allocator = hybrid 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore allocator = hybrid 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore zero block detection = True 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 20 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 20 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-16T04:06:41.738 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 10 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: osd blocked scrub grace period = 3600 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug reject backfill probability = 0.1 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify cached snaps = True 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify missing on start = True 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: osd max backfills = 6 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: osd max markdown count = 1000 2024-09-16T04:06:41.739 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock override recovery settings = True 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: osd op queue = debug_random 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: osd op queue cut off = debug_random 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub min interval = 60 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-16T04:06:41.740 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:06:41.741 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:06:41.742 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: mon scrub interval = 300 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:06:41.743 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:06:41.744 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-09-16T04:06:41.744 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-09-16T04:06:41.744 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = cephx 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.745 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.746 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:06:41.747 INFO:teuthology.orchestra.run.smithi174.stdout: mon_allow_pool_size_one = true 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = "172.21.15.16,172.21.15.174,172.21.15.194" 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: mon client directed command retry = 5 2024-09-16T04:06:41.748 INFO:teuthology.orchestra.run.smithi174.stdout: mon election default strategy = 3 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay max = 1 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay probability = 0.005 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject delay type = osd 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject internal delays = 0.002 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: ms inject socket failures = 2500 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = 44381953-4509-4b15-853e-3887928dfa29 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.749 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 120 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug trim objects = true 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2024-09-16T04:06:41.750 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: debug monc = 20 2024-09-16T04:06:41.751 INFO:teuthology.orchestra.run.smithi174.stdout: bluefs allocator = hybrid 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore allocator = hybrid 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore block size = 96636764160 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore fsck on mount = True 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore zero block detection = True 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluefs = 20 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluestore = 20 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 20 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: debug rocksdb = 10 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd full ratio = 0.9 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:06:41.752 INFO:teuthology.orchestra.run.smithi174.stdout: osd blocked scrub grace period = 3600 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug reject backfill probability = 0.1 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify cached snaps = True 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify missing on start = True 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd max backfills = 6 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd max markdown count = 1000 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock override recovery settings = True 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = bluestore 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd op queue = debug_random 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd op queue cut off = debug_random 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub min interval = 60 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2024-09-16T04:06:41.753 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 20 2024-09-16T04:06:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:06:41.755 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: # 1m isn't quite enough 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: mon scrub interval = 300 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2024-09-16T04:06:41.756 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.b] 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.c] 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi194.stdout:[global] 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi194.stdout: chdir = "" 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi194.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi194.stdout: auth supported = cephx 2024-09-16T04:06:41.757 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: filestore xattr use omap = true 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: mon clock drift allowed = 1.000 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: osd crush chooseleaf type = 0 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: auth debug = true 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on old message = true 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on bug = true 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:06:41.758 INFO:teuthology.orchestra.run.smithi194.stdout: mon pg warn max object skew = 0 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default size = 2 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow primary affinity = true 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow pg remap = true 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on no sortbitwise = false 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:06:41.760 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on too few osds = false 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: mon_allow_pool_size_one = true 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: osd default data pool replay window = 5 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: mon allow pool delete = true 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: mon cluster log file level = debug 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: debug asserts on shutdown = true 2024-09-16T04:06:41.761 INFO:teuthology.orchestra.run.smithi194.stdout: mon health detail to clog = false 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: mon host = "172.21.15.16,172.21.15.174,172.21.15.194" 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: mon client directed command retry = 5 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: mon election default strategy = 3 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: ms inject delay max = 1 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: ms inject delay probability = 0.005 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: ms inject delay type = osd 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: ms inject internal delays = 0.002 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: ms inject socket failures = 2500 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: fsid = 44381953-4509-4b15-853e-3887928dfa29 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout:[osd] 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: osd journal size = 100 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.762 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub load threshold = 5.0 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub max interval = 120 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: osd recover clone overlap = true 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: osd recovery max chunk = 1048576 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug shutdown = true 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug op order = true 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify stray on activate = true 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug trim objects = true 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.763 INFO:teuthology.orchestra.run.smithi194.stdout: osd open classes on start = true 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug pg log writeout = true 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: osd map max advance = 10 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: journal zero on create = true 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: filestore apply finisher threads = 3 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: bdev debug aio = true 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug misdirected ops = true 2024-09-16T04:06:41.764 INFO:teuthology.orchestra.run.smithi194.stdout: debug monc = 20 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: bluefs allocator = hybrid 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore allocator = hybrid 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore block size = 96636764160 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore fsck on mount = True 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore zero block detection = True 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluefs = 20 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluestore = 20 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: debug osd = 20 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: debug rocksdb = 10 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd full ratio = 0.9 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:06:41.765 INFO:teuthology.orchestra.run.smithi194.stdout: osd blocked scrub grace period = 3600 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug reject backfill probability = 0.1 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify cached snaps = True 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify missing on start = True 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd max backfills = 6 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd max markdown count = 1000 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock override recovery settings = True 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd objectstore = bluestore 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd op queue = debug_random 2024-09-16T04:06:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: osd op queue cut off = debug_random 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub min interval = 60 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr] 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgr = 20 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: mgr/telemetry/nag = false 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout:[mon] 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-09-16T04:06:41.767 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-09-16T04:06:41.768 INFO:teuthology.orchestra.run.smithi194.stdout: debug paxos = 20 2024-09-16T04:06:41.768 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-09-16T04:06:41.768 INFO:teuthology.orchestra.run.smithi194.stdout: mon data avail warn = 5 2024-09-16T04:06:41.768 INFO:teuthology.orchestra.run.smithi194.stdout: mon mgr mkfs grace = 240 2024-09-16T04:06:41.768 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:06:41.768 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd reporter subtree level = osd 2024-09-16T04:06:41.768 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd prime pg temp = true 2024-09-16T04:06:41.768 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:06:41.768 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: # 1m isn't quite enough 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:06:41.769 INFO:teuthology.orchestra.run.smithi194.stdout: mon scrub interval = 300 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout:[client] 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout: rgw cache enabled = true 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable ops log = true 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable usage log = true 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.a] 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.b] 2024-09-16T04:06:41.770 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.c] 2024-09-16T04:06:41.772 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T04:06:41.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T04:06:41.867 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T04:06:41.868 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:41.868 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T04:06:41.883 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:41.883 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T04:06:41.939 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:06:41.940 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:41.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:06:41.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:06:42.015 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:06:42.016 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:42.071 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:06:42.072 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:06:42.072 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:06:42.072 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:06:42.113 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:06:42.113 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:42.171 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:06:42.171 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:06:42.171 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:06:42.171 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:06:42.206 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:06:42.206 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:42.262 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T04:06:42.262 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T04:06:42.262 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:06:42.349 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:06:42.352 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T04:06:42.352 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T04:06:42.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T04:06:42.395 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T04:06:42.408 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T04:06:42.409 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'}} 2024-09-16T04:06:42.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T04:06:42.474 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:42.475 INFO:tasks.ceph:role: osd.0 2024-09-16T04:06:42.475 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:06:42.475 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:42.583 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-16T04:06:42.584 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-16T04:06:42.585 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T04:06:42.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T04:06:42.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T04:06:42.792 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:42.792 INFO:tasks.ceph:role: osd.1 2024-09-16T04:06:42.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:06:42.792 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:06:42.856 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:42.895 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-16T04:06:42.897 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-16T04:06:42.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T04:06:42.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T04:06:43.039 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T04:06:43.106 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:43.106 INFO:tasks.ceph:role: osd.2 2024-09-16T04:06:43.106 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:06:43.106 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:06:43.172 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:43.172 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:43.173 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:43.173 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:43.173 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:43.173 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:43.173 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:43.211 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-16T04:06:43.213 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-16T04:06:43.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T04:06:43.284 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T04:06:43.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T04:06:43.420 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:43.420 INFO:tasks.ceph:role: osd.3 2024-09-16T04:06:43.420 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:06:43.420 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:06:43.486 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:43.487 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:43.487 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:43.487 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:43.487 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:43.525 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-16T04:06:43.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-16T04:06:43.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-16T04:06:43.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T04:06:43.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:43.764+0000 7f479002c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T04:06:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:43.764+0000 7f479002c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T04:06:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:43.764+0000 7f479002c640 -1 bdev(0x55a31f027800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T04:06:43.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:43.764+0000 7f479002c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T04:06:47.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T04:06:47.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:47.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:47.542+0000 7fbde39c5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T04:06:47.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:47.543+0000 7fbde39c5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T04:06:47.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:47.543+0000 7fbde39c5640 -1 bdev(0x559371a07800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T04:06:47.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:47.543+0000 7fbde39c5640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T04:06:50.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T04:06:50.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:50.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:50.822+0000 7f23d4c18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T04:06:50.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:50.823+0000 7f23d4c18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T04:06:50.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:50.823+0000 7f23d4c18640 -1 bdev(0x55b8ca239800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T04:06:50.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:50.823+0000 7f23d4c18640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T04:06:53.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T04:06:53.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:53.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:53.866+0000 7f6f0a102640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T04:06:53.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:53.866+0000 7f6f0a102640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T04:06:53.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:53.866+0000 7f6f0a102640 -1 bdev(0x55c36ff8b800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T04:06:53.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:06:53.866+0000 7f6f0a102640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T04:06:56.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T04:06:56.804 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T04:06:56.835 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:56.835 INFO:tasks.ceph:role: osd.4 2024-09-16T04:06:56.835 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:06:56.835 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:06:56.911 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:56.912 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:56.912 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:56.912 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:56.912 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:56.912 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:56.912 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:56.912 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:56.913 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:56.913 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:57.407 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-09-16T04:06:57.409 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-16T04:06:57.409 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-16T04:06:57.547 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T04:06:57.577 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T04:06:57.643 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:57.643 INFO:tasks.ceph:role: osd.5 2024-09-16T04:06:57.643 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:06:57.644 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:06:57.717 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:57.717 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:57.717 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:57.717 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:57.717 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:57.717 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:57.718 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:57.718 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:57.718 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:57.718 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:58.164 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-09-16T04:06:58.167 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-16T04:06:58.167 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T04:06:58.253 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T04:06:58.282 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T04:06:58.347 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:58.347 INFO:tasks.ceph:role: osd.6 2024-09-16T04:06:58.347 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:06:58.348 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:06:58.420 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:58.420 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:58.420 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:58.421 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:58.421 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:58.421 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:58.421 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:58.421 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:58.421 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:58.421 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:58.907 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-09-16T04:06:58.909 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-16T04:06:58.909 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-16T04:06:59.051 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T04:06:59.081 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T04:06:59.146 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T04:06:59.146 INFO:tasks.ceph:role: osd.7 2024-09-16T04:06:59.146 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:06:59.147 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:06:59.220 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:59.220 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:59.220 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:59.220 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:59.220 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:59.220 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:59.220 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:59.221 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:59.221 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:59.221 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:59.730 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-09-16T04:06:59.732 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-16T04:06:59.732 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-16T04:06:59.869 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T04:06:59.902 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:00.001 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:06:59.998+0000 7fccb353e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T04:07:00.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:00.000+0000 7fccb353e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T04:07:00.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:00.000+0000 7fccb353e640 -1 bdev(0x561d41eeb800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T04:07:00.002 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:00.000+0000 7fccb353e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T04:07:02.917 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T04:07:02.945 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:03.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:03.031+0000 7fbc49afe640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T04:07:03.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:03.032+0000 7fbc49afe640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T04:07:03.035 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:03.032+0000 7fbc49afe640 -1 bdev(0x55b94799b800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T04:07:03.035 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:03.032+0000 7fbc49afe640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T04:07:05.950 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T04:07:05.979 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:06.070 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:06.067+0000 7f079bad8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T04:07:06.070 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:06.067+0000 7f079bad8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T04:07:06.070 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:06.067+0000 7f079bad8640 -1 bdev(0x562a3f1d9800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T04:07:06.070 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:06.067+0000 7f079bad8640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T04:07:09.226 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T04:07:09.256 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:09.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:09.343+0000 7f765d61c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T04:07:09.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:09.343+0000 7f765d61c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T04:07:09.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:09.343+0000 7f765d61c640 -1 bdev(0x55c528add800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T04:07:09.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-16T04:07:09.343+0000 7f765d61c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T04:07:12.271 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T04:07:12.300 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-16T04:07:12.331 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-16T04:07:12.331 INFO:tasks.ceph:role: osd.8 2024-09-16T04:07:12.331 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:07:12.331 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:07:12.406 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:07:12.903 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-09-16T04:07:12.905 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-09-16T04:07:12.906 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-8 2024-09-16T04:07:13.042 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-16T04:07:13.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-16T04:07:13.138 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-16T04:07:13.138 INFO:tasks.ceph:role: osd.9 2024-09-16T04:07:13.138 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:07:13.138 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:07:13.209 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:07:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:07:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:07:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:07:13.210 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:07:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:07:13.210 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:07:13.210 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:07:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:07:13.210 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:07:13.663 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-09-16T04:07:13.665 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-09-16T04:07:13.666 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-9 2024-09-16T04:07:13.746 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-16T04:07:13.815 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-16T04:07:13.881 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-16T04:07:13.881 INFO:tasks.ceph:role: osd.10 2024-09-16T04:07:13.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:07:13.881 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:07:13.953 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:07:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:07:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:07:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:07:13.953 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:07:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:07:13.953 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:07:13.953 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:07:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:07:13.954 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:07:14.465 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-09-16T04:07:14.467 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-09-16T04:07:14.467 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-16T04:07:14.592 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-16T04:07:14.625 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-16T04:07:14.691 INFO:tasks.ceph:roles_to_devs: {'osd.8': '/dev/vg_nvme/lv_1', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.11': '/dev/vg_nvme/lv_4'} 2024-09-16T04:07:14.691 INFO:tasks.ceph:role: osd.11 2024-09-16T04:07:14.691 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:07:14.691 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T04:07:14.764 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:07:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:07:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:07:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:07:14.764 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:07:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:07:14.764 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:07:14.764 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:07:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:07:14.765 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:07:15.267 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-09-16T04:07:15.269 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-09-16T04:07:15.270 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-11 2024-09-16T04:07:15.411 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-16T04:07:15.444 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:15.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:15.551+0000 7f3b0af1d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-16T04:07:15.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:15.551+0000 7f3b0af1d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-16T04:07:15.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:15.552+0000 7f3b0af1d640 -1 bdev(0x55de2fe43800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-16T04:07:15.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:15.552+0000 7f3b0af1d640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-16T04:07:18.474 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-16T04:07:18.503 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:18.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:18.592+0000 7f718dbbb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-16T04:07:18.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:18.592+0000 7f718dbbb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-16T04:07:18.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:18.592+0000 7f718dbbb640 -1 bdev(0x5587bb95b800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-16T04:07:18.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:18.592+0000 7f718dbbb640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-16T04:07:21.763 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-16T04:07:21.792 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:21.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:21.881+0000 7fac0ed54640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-16T04:07:21.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:21.881+0000 7fac0ed54640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-16T04:07:21.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:21.881+0000 7fac0ed54640 -1 bdev(0x5610bc48d800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-16T04:07:21.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:21.882+0000 7fac0ed54640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-16T04:07:25.057 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-16T04:07:25.087 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:25.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:25.174+0000 7f4804c12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-16T04:07:25.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:25.175+0000 7f4804c12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-16T04:07:25.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:25.175+0000 7f4804c12640 -1 bdev(0x55ba0aacd800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-16T04:07:25.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-16T04:07:25.175+0000 7f4804c12640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-16T04:07:28.095 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-16T04:07:28.124 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T04:07:28.124 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:07:28.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T04:07:28.157 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:07:28.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T04:07:28.227 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:07:28.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T04:07:28.297 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:07:28.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T04:07:28.367 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:07:28.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T04:07:28.398 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:07:28.398 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T04:07:28.468 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:07:28.468 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T04:07:28.537 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:07:28.537 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T04:07:28.606 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:07:28.606 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T04:07:28.673 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:07:28.673 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-16T04:07:28.703 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:07:28.703 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-16T04:07:28.773 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:07:28.773 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-16T04:07:28.842 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:07:28.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-16T04:07:28.911 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:07:28.911 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T04:07:28.931 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T04:07:28.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:07:28.974 DEBUG:teuthology.orchestra.run.smithi174:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:07:28.976 DEBUG:teuthology.orchestra.run.smithi194:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:07:29.000 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2024-09-16T04:07:29.000 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDTrudmXgWbLRAAUxG1aPo3SsFO37oEYP4/TA== 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDXrudmTR9rIBAAsFZBIiHJXV+ZdrYUgTwhOw== 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDarudmv/IZMRAANXWXywzV1qszDwyVhEYuYg== 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDdrudmeou1MxAA1fwqLeelXh/bYGmCDB/u8w== 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDSrudmuM/KFBAA8I8uvxVVx4RMgwtKzxquRw== 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.4] 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDkrudmvv4EABAAj+1H9CF2mqeL1KuuwoDYIA== 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.5] 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDnrudmHIz9ARAAMGQ6ZsEf+pgZHmXSHjmNuA== 2024-09-16T04:07:29.001 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.6] 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDqrudmzWAaBBAAXUxWBq0DlLrTBr9lOtQx6w== 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.7] 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDtrudmFSiRFBAAyDHo08D/P63YeTd/6nuGBQ== 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.8] 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDzrudm1ejuIBAACozH91YFxHdiViBlxJYSUA== 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.9] 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQD2rudmyABcIxAA0IdhfXxnvvkYaR8Z0Ts0JQ== 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.10] 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQD5rudmAweZNBAAG8yXrxnYMxuCAGahOYJvWA== 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.11] 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQD9rudmBkh4ChAARVx0QAJcFSEKeprOpCTxnw== 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-09-16T04:07:29.002 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQDSrudmlg2FFxAA8uTYW3jOdU04HvVoqX/AbA== 2024-09-16T04:07:29.005 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.0] 2024-09-16T04:07:29.005 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDTrudmXgWbLRAAUxG1aPo3SsFO37oEYP4/TA== 2024-09-16T04:07:29.005 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.1] 2024-09-16T04:07:29.006 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDXrudmTR9rIBAAsFZBIiHJXV+ZdrYUgTwhOw== 2024-09-16T04:07:29.006 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.2] 2024-09-16T04:07:29.006 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDarudmv/IZMRAANXWXywzV1qszDwyVhEYuYg== 2024-09-16T04:07:29.006 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.3] 2024-09-16T04:07:29.006 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDdrudmeou1MxAA1fwqLeelXh/bYGmCDB/u8w== 2024-09-16T04:07:29.006 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.x] 2024-09-16T04:07:29.006 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDSrudmuM/KFBAA8I8uvxVVx4RMgwtKzxquRw== 2024-09-16T04:07:29.006 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.4] 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDkrudmvv4EABAAj+1H9CF2mqeL1KuuwoDYIA== 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.5] 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDnrudmHIz9ARAAMGQ6ZsEf+pgZHmXSHjmNuA== 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.6] 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDqrudmzWAaBBAAXUxWBq0DlLrTBr9lOtQx6w== 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.7] 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDtrudmFSiRFBAAyDHo08D/P63YeTd/6nuGBQ== 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.8] 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDzrudm1ejuIBAACozH91YFxHdiViBlxJYSUA== 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.9] 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD2rudmyABcIxAA0IdhfXxnvvkYaR8Z0Ts0JQ== 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.10] 2024-09-16T04:07:29.007 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD5rudmAweZNBAAG8yXrxnYMxuCAGahOYJvWA== 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.11] 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQD9rudmBkh4ChAARVx0QAJcFSEKeprOpCTxnw== 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi194.stdout:[client.0] 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDSrudmlg2FFxAA8uTYW3jOdU04HvVoqX/AbA== 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.0] 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDTrudmXgWbLRAAUxG1aPo3SsFO37oEYP4/TA== 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.1] 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDXrudmTR9rIBAAsFZBIiHJXV+ZdrYUgTwhOw== 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.2] 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDarudmv/IZMRAANXWXywzV1qszDwyVhEYuYg== 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.3] 2024-09-16T04:07:29.008 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDdrudmeou1MxAA1fwqLeelXh/bYGmCDB/u8w== 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.x] 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDSrudmuM/KFBAA8I8uvxVVx4RMgwtKzxquRw== 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.4] 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDkrudmvv4EABAAj+1H9CF2mqeL1KuuwoDYIA== 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.5] 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDnrudmHIz9ARAAMGQ6ZsEf+pgZHmXSHjmNuA== 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.6] 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDqrudmzWAaBBAAXUxWBq0DlLrTBr9lOtQx6w== 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.7] 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDtrudmFSiRFBAAyDHo08D/P63YeTd/6nuGBQ== 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.8] 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDzrudm1ejuIBAACozH91YFxHdiViBlxJYSUA== 2024-09-16T04:07:29.009 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.9] 2024-09-16T04:07:29.010 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQD2rudmyABcIxAA0IdhfXxnvvkYaR8Z0Ts0JQ== 2024-09-16T04:07:29.010 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.10] 2024-09-16T04:07:29.010 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQD5rudmAweZNBAAG8yXrxnYMxuCAGahOYJvWA== 2024-09-16T04:07:29.010 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.11] 2024-09-16T04:07:29.010 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQD9rudmBkh4ChAARVx0QAJcFSEKeprOpCTxnw== 2024-09-16T04:07:29.010 INFO:teuthology.orchestra.run.smithi174.stdout:[client.0] 2024-09-16T04:07:29.010 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDSrudmlg2FFxAA8uTYW3jOdU04HvVoqX/AbA== 2024-09-16T04:07:29.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.048 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.110 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.113 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.167 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.170 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.223 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.225 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:07:29.279 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:07:29.282 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:07:29.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.334 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.336 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.389 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.391 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.448 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.501 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.504 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.557 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.560 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.613 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.615 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.666 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.668 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.670 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.723 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.726 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:07:29.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:07:29.778 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:07:29.782 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:07:29.835 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T04:07:29.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T04:07:29.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:07:29.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T04:07:29.985 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T04:07:30.010 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:07:30.107 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T04:07:30.131 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T04:07:30.157 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:07:30.254 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T04:07:30.280 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:30.283 DEBUG:teuthology.orchestra.run.smithi174:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:30.285 DEBUG:teuthology.orchestra.run.smithi194:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:07:30.336 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T04:07:30.336 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:07:30.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:07:30.339 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:07:30.339 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:07:30.339 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:07:30.343 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:07:30.344 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:07:30.344 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:07:30.378 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:07:30.378 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T04:07:30.378 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T04:07:30.378 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T04:07:30.381 INFO:tasks.ceph.mgr.x:Started 2024-09-16T04:07:30.381 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T04:07:30.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T04:07:36.770 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-09-16T04:07:36.780 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:36.779+0000 7fe6c2721040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T04:07:36.782 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T04:07:36.782 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T04:07:36.782 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:07:36.783 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T04:07:36.809 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:07:36.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T04:07:36.821 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:36.819+0000 7fe6c2721040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T04:07:36.874 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:07:36.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T04:07:36.941 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-16T04:07:36.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T04:07:36.962 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:36.961+0000 7fe6c2721040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T04:07:37.006 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:07:37.006 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T04:07:37.013 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:37.011+0000 7fe6c2721040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T04:07:37.031 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:07:37.031 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T04:07:37.097 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:07:37.097 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T04:07:37.162 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-16T04:07:37.163 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T04:07:37.184 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:37.182+0000 7fe6c2721040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:07:37.208 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:07:37.208 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-16T04:07:37.239 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:07:37.239 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-16T04:07:37.297 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:37.295+0000 7fe6c2721040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T04:07:37.307 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:07:37.307 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-16T04:07:37.375 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-16T04:07:37.375 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-16T04:07:37.442 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 4506422f-7ccf-46d7-af66-62e8cace167a 0 2024-09-16T04:07:37.607 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:37.605+0000 7fe6c2721040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T04:07:37.660 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:37.659+0000 7fe6c2721040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T04:07:37.669 INFO:teuthology.orchestra.run.smithi194.stdout:0 2024-09-16T04:07:37.682 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 8dc0488e-efb6-4664-a799-5e14c56769b0 1 2024-09-16T04:07:37.809 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:37.807+0000 7fe6c2721040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T04:07:37.865 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-09-16T04:07:37.876 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 1686e657-c1f3-499b-8b10-67f428c5c216 2 2024-09-16T04:07:38.061 INFO:teuthology.orchestra.run.smithi194.stdout:2 2024-09-16T04:07:38.072 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 730ad37c-1f5f-4d23-a1a7-c75320506665 3 2024-09-16T04:07:38.251 INFO:teuthology.orchestra.run.smithi194.stdout:3 2024-09-16T04:07:38.262 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new d82c5c44-72fe-4680-9c04-ab4e0775b929 4 2024-09-16T04:07:38.446 INFO:teuthology.orchestra.run.smithi194.stdout:4 2024-09-16T04:07:38.458 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 41502441-b0d9-460f-92c0-b8b3a915cac5 5 2024-09-16T04:07:38.638 INFO:teuthology.orchestra.run.smithi194.stdout:5 2024-09-16T04:07:38.649 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 45cc7521-11db-4496-8d0c-1abc42962810 6 2024-09-16T04:07:38.758 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:38.757+0000 7fe6c2721040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T04:07:38.810 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:38.808+0000 7fe6c2721040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T04:07:38.833 INFO:teuthology.orchestra.run.smithi194.stdout:6 2024-09-16T04:07:38.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 1b5d6337-73c3-494b-91cd-fcfa7d307050 7 2024-09-16T04:07:39.026 INFO:teuthology.orchestra.run.smithi194.stdout:7 2024-09-16T04:07:39.038 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 766efffa-3aff-4aa3-9cf8-88c5c7986d21 8 2024-09-16T04:07:39.174 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.172+0000 7fe6c2721040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T04:07:39.219 INFO:teuthology.orchestra.run.smithi194.stdout:8 2024-09-16T04:07:39.230 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 755cc9aa-2f6f-4d7a-84cf-7feb74d65c48 9 2024-09-16T04:07:39.259 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.257+0000 7fe6c2721040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T04:07:39.365 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.363+0000 7fe6c2721040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:07:39.409 INFO:teuthology.orchestra.run.smithi194.stdout:9 2024-09-16T04:07:39.410 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.409+0000 7fe6c2721040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T04:07:39.421 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377 10 2024-09-16T04:07:39.524 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.522+0000 7fe6c2721040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T04:07:39.577 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.575+0000 7fe6c2721040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T04:07:39.602 INFO:teuthology.orchestra.run.smithi194.stdout:10 2024-09-16T04:07:39.613 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new f71641eb-1331-4b39-9c0e-bf98e91ed295 11 2024-09-16T04:07:39.622 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.620+0000 7fe6c2721040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T04:07:39.734 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.732+0000 7fe6c2721040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T04:07:39.783 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.782+0000 7fe6c2721040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T04:07:39.797 INFO:teuthology.orchestra.run.smithi194.stdout:11 2024-09-16T04:07:39.808 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T04:07:39.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T04:07:39.812 INFO:tasks.ceph.osd.0:Started 2024-09-16T04:07:39.812 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T04:07:39.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T04:07:39.814 INFO:tasks.ceph.osd.1:Started 2024-09-16T04:07:39.814 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T04:07:39.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T04:07:39.817 INFO:tasks.ceph.osd.2:Started 2024-09-16T04:07:39.817 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T04:07:39.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T04:07:39.819 INFO:tasks.ceph.osd.3:Started 2024-09-16T04:07:39.819 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T04:07:39.819 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T04:07:39.821 INFO:tasks.ceph.osd.4:Started 2024-09-16T04:07:39.822 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T04:07:39.822 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T04:07:39.824 INFO:tasks.ceph.osd.5:Started 2024-09-16T04:07:39.824 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T04:07:39.824 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T04:07:39.826 INFO:tasks.ceph.osd.6:Started 2024-09-16T04:07:39.826 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T04:07:39.826 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T04:07:39.828 INFO:tasks.ceph.osd.7:Started 2024-09-16T04:07:39.829 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-16T04:07:39.829 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-16T04:07:39.831 INFO:tasks.ceph.osd.8:Started 2024-09-16T04:07:39.831 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-16T04:07:39.831 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-16T04:07:39.833 INFO:tasks.ceph.osd.9:Started 2024-09-16T04:07:39.833 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-16T04:07:39.833 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-16T04:07:39.835 INFO:tasks.ceph.osd.10:Started 2024-09-16T04:07:39.835 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-16T04:07:39.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-16T04:07:39.837 INFO:tasks.ceph.osd.11:Started 2024-09-16T04:07:39.837 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:07:39.840 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.839+0000 7fe6c2721040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T04:07:39.920 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.919+0000 7fe6c2721040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T04:07:39.966 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:39.965+0000 7fe6c2721040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T04:07:40.011 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:40.009+0000 7fe6c2721040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T04:07:40.041 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:07:40.042 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"fsid":"44381953-4509-4b15-853e-3887928dfa29","created":"2024-09-16T04:07:36.650572+0000","modified":"2024-09-16T04:07:39.793439+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:07:39.793439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4506422f-7ccf-46d7-af66-62e8cace167a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dc0488e-efb6-4664-a799-5e14c56769b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1686e657-c1f3-499b-8b10-67f428c5c216","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"730ad37c-1f5f-4d23-a1a7-c75320506665","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d82c5c44-72fe-4680-9c04-ab4e0775b929","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"41502441-b0d9-460f-92c0-b8b3a915cac5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"45cc7521-11db-4496-8d0c-1abc42962810","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1b5d6337-73c3-494b-91cd-fcfa7d307050","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"766efffa-3aff-4aa3-9cf8-88c5c7986d21","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"755cc9aa-2f6f-4d7a-84cf-7feb74d65c48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"f71641eb-1331-4b39-9c0e-bf98e91ed295","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:07:40.052 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T04:07:40.053 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T04:07:40.662 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:40.661+0000 7fe6c2721040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T04:07:41.060 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:07:41.058+0000 7f46fcb1e640 -1 Falling back to public interface 2024-09-16T04:07:41.065 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:07:41.064+0000 7f8f3b015640 -1 Falling back to public interface 2024-09-16T04:07:41.066 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:07:41.064+0000 7f3072927640 -1 Falling back to public interface 2024-09-16T04:07:41.264 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:07:41.295 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:07:41.293+0000 7fe1d1ae0640 -1 Falling back to public interface 2024-09-16T04:07:41.296 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:07:41.294+0000 7f9d73b2e640 -1 Falling back to public interface 2024-09-16T04:07:41.297 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:07:41.295+0000 7f3197210640 -1 Falling back to public interface 2024-09-16T04:07:41.299 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:07:41.297+0000 7f3011b20640 -1 Falling back to public interface 2024-09-16T04:07:41.300 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:07:41.298+0000 7eff4efb0640 -1 Falling back to public interface 2024-09-16T04:07:41.318 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:07:41.316+0000 7ffa98020640 -1 Falling back to public interface 2024-09-16T04:07:41.444 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-16T04:07:41.444 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":14,"fsid":"44381953-4509-4b15-853e-3887928dfa29","created":"2024-09-16T04:07:36.650572+0000","modified":"2024-09-16T04:07:39.793439+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:07:39.793439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4506422f-7ccf-46d7-af66-62e8cace167a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dc0488e-efb6-4664-a799-5e14c56769b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1686e657-c1f3-499b-8b10-67f428c5c216","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"730ad37c-1f5f-4d23-a1a7-c75320506665","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d82c5c44-72fe-4680-9c04-ab4e0775b929","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"41502441-b0d9-460f-92c0-b8b3a915cac5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"45cc7521-11db-4496-8d0c-1abc42962810","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1b5d6337-73c3-494b-91cd-fcfa7d307050","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"766efffa-3aff-4aa3-9cf8-88c5c7986d21","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"755cc9aa-2f6f-4d7a-84cf-7feb74d65c48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"f71641eb-1331-4b39-9c0e-bf98e91ed295","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:07:41.453 DEBUG:teuthology.misc:0 of 12 OSDs are up 2024-09-16T04:07:41.522 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:07:41.520+0000 7f37de460640 -1 Falling back to public interface 2024-09-16T04:07:41.540 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:07:41.538+0000 7f652bba8640 -1 Falling back to public interface 2024-09-16T04:07:41.682 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:41.681+0000 7fe6626d5640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T04:07:41.687 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-16T04:07:41.685+0000 7fe6626d5640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T04:07:41.808 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:07:41.806+0000 7f82d8d58640 -1 Falling back to public interface 2024-09-16T04:07:44.839 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:07:44.837+0000 7ffa98020640 -1 osd.0 0 log_to_monitors true 2024-09-16T04:07:44.892 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:07:44.890+0000 7f3072927640 -1 osd.3 0 log_to_monitors true 2024-09-16T04:07:44.916 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:07:44.914+0000 7f8f3b015640 -1 osd.1 0 log_to_monitors true 2024-09-16T04:07:45.110 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:07:45.107+0000 7eff4efb0640 -1 osd.9 0 log_to_monitors true 2024-09-16T04:07:45.127 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:07:45.125+0000 7f46fcb1e640 -1 osd.2 0 log_to_monitors true 2024-09-16T04:07:45.133 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:07:45.131+0000 7fe1d1ae0640 -1 osd.6 0 log_to_monitors true 2024-09-16T04:07:45.138 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:07:45.135+0000 7f3197210640 -1 osd.11 0 log_to_monitors true 2024-09-16T04:07:45.141 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:07:45.139+0000 7f3011b20640 -1 osd.4 0 log_to_monitors true 2024-09-16T04:07:45.271 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:07:45.268+0000 7f37de460640 -1 osd.10 0 log_to_monitors true 2024-09-16T04:07:45.573 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:07:45.570+0000 7f9d73b2e640 -1 osd.8 0 log_to_monitors true 2024-09-16T04:07:45.609 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:07:45.607+0000 7f652bba8640 -1 osd.7 0 log_to_monitors true 2024-09-16T04:07:46.114 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:07:46.113+0000 7f82d8d58640 -1 osd.5 0 log_to_monitors true 2024-09-16T04:07:46.563 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:07:46.561+0000 7ffa93911640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T04:07:46.576 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:07:46.574+0000 7ffa8b731640 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:46.917 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:07:46.914+0000 7eff4af3f640 -1 osd.9 0 waiting for initial osdmap 2024-09-16T04:07:46.917 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:07:46.914+0000 7f37da3ef640 -1 osd.10 0 waiting for initial osdmap 2024-09-16T04:07:46.917 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:07:46.915+0000 7f9d6fabd640 -1 osd.8 0 waiting for initial osdmap 2024-09-16T04:07:46.919 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:07:46.918+0000 7f8f368f5640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T04:07:46.927 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:07:46.925+0000 7f8f3271d640 -1 osd.1 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:46.936 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:07:46.933+0000 7f9d6b8e5640 -1 osd.8 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:46.941 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:07:46.939+0000 7f37d6217640 -1 osd.10 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:46.942 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:07:46.939+0000 7eff46d67640 -1 osd.9 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:46.945 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:07:46.943+0000 7f6527b37640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T04:07:46.984 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:07:46.982+0000 7f652395f640 -1 osd.7 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:47.701 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:07:47.698+0000 7f3192a97640 -1 osd.11 0 waiting for initial osdmap 2024-09-16T04:07:47.717 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:07:47.714+0000 7f318a8b7640 -1 osd.11 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:47.905 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:07:47.899+0000 7f306f0ca640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T04:07:47.905 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:07:47.902+0000 7f46f8aad640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T04:07:47.919 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:07:47.917+0000 7f46f08cd640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:47.919 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:07:47.917+0000 7f30666d6640 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:48.141 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:07:48.139+0000 7f300daaf640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T04:07:48.161 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:07:48.160+0000 7f30058cf640 -1 osd.4 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:48.189 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:07:48.188+0000 7fe1cda6f640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T04:07:48.216 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:07:48.215+0000 7fe1c588f640 -1 osd.6 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:48.665 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:07:48.730 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:07:48.729+0000 7f82d4ce7640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T04:07:48.734 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:07:48.732+0000 7f82ccb07640 -1 osd.5 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:07:48.973 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-16T04:07:48.973 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":19,"fsid":"44381953-4509-4b15-853e-3887928dfa29","created":"2024-09-16T04:07:36.650572+0000","modified":"2024-09-16T04:07:48.919317+0000","last_up_change":"2024-09-16T04:07:48.919317+0000","last_in_change":"2024-09-16T04:07:39.793439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:07:48.723300+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4506422f-7ccf-46d7-af66-62e8cace167a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6825","nonce":3110165165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6827","nonce":3110165165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6831","nonce":3110165165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6829","nonce":3110165165}]},"public_addr":"172.21.15.16:6825/3110165165","cluster_addr":"172.21.15.16:6827/3110165165","heartbeat_back_addr":"172.21.15.16:6831/3110165165","heartbeat_front_addr":"172.21.15.16:6829/3110165165","state":["exists","up"]},{"osd":1,"uuid":"8dc0488e-efb6-4664-a799-5e14c56769b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6810","nonce":3040423711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6814","nonce":3040423711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6822","nonce":3040423711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6818","nonce":3040423711}]},"public_addr":"172.21.15.16:6810/3040423711","cluster_addr":"172.21.15.16:6814/3040423711","heartbeat_back_addr":"172.21.15.16:6822/3040423711","heartbeat_front_addr":"172.21.15.16:6818/3040423711","state":["exists","up"]},{"osd":2,"uuid":"1686e657-c1f3-499b-8b10-67f428c5c216","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6801","nonce":2266032822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6803","nonce":2266032822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6807","nonce":2266032822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6805","nonce":2266032822}]},"public_addr":"172.21.15.16:6801/2266032822","cluster_addr":"172.21.15.16:6803/2266032822","heartbeat_back_addr":"172.21.15.16:6807/2266032822","heartbeat_front_addr":"172.21.15.16:6805/2266032822","state":["exists","up"]},{"osd":3,"uuid":"730ad37c-1f5f-4d23-a1a7-c75320506665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6811","nonce":3389562884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6815","nonce":3389562884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6823","nonce":3389562884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6819","nonce":3389562884}]},"public_addr":"172.21.15.16:6811/3389562884","cluster_addr":"172.21.15.16:6815/3389562884","heartbeat_back_addr":"172.21.15.16:6823/3389562884","heartbeat_front_addr":"172.21.15.16:6819/3389562884","state":["exists","up"]},{"osd":4,"uuid":"d82c5c44-72fe-4680-9c04-ab4e0775b929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6811","nonce":1733996396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6813","nonce":1733996396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6817","nonce":1733996396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6815","nonce":1733996396}]},"public_addr":"172.21.15.174:6811/1733996396","cluster_addr":"172.21.15.174:6813/1733996396","heartbeat_back_addr":"172.21.15.174:6817/1733996396","heartbeat_front_addr":"172.21.15.174:6815/1733996396","state":["exists","up"]},{"osd":5,"uuid":"41502441-b0d9-460f-92c0-b8b3a915cac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6827","nonce":2956753716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6829","nonce":2956753716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6833","nonce":2956753716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6831","nonce":2956753716}]},"public_addr":"172.21.15.174:6827/2956753716","cluster_addr":"172.21.15.174:6829/2956753716","heartbeat_back_addr":"172.21.15.174:6833/2956753716","heartbeat_front_addr":"172.21.15.174:6831/2956753716","state":["exists","up"]},{"osd":6,"uuid":"45cc7521-11db-4496-8d0c-1abc42962810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6803","nonce":2727832023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6805","nonce":2727832023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6809","nonce":2727832023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6807","nonce":2727832023}]},"public_addr":"172.21.15.174:6803/2727832023","cluster_addr":"172.21.15.174:6805/2727832023","heartbeat_back_addr":"172.21.15.174:6809/2727832023","heartbeat_front_addr":"172.21.15.174:6807/2727832023","state":["exists","up"]},{"osd":7,"uuid":"1b5d6337-73c3-494b-91cd-fcfa7d307050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6819","nonce":2338392542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6821","nonce":2338392542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6825","nonce":2338392542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6823","nonce":2338392542}]},"public_addr":"172.21.15.174:6819/2338392542","cluster_addr":"172.21.15.174:6821/2338392542","heartbeat_back_addr":"172.21.15.174:6825/2338392542","heartbeat_front_addr":"172.21.15.174:6823/2338392542","state":["exists","up"]},{"osd":8,"uuid":"766efffa-3aff-4aa3-9cf8-88c5c7986d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621487148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621487148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621487148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621487148}]},"public_addr":"172.21.15.194:6801/2621487148","cluster_addr":"172.21.15.194:6803/2621487148","heartbeat_back_addr":"172.21.15.194:6807/2621487148","heartbeat_front_addr":"172.21.15.194:6805/2621487148","state":["exists","up"]},{"osd":9,"uuid":"755cc9aa-2f6f-4d7a-84cf-7feb74d65c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6817","nonce":367304498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6819","nonce":367304498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6823","nonce":367304498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6821","nonce":367304498}]},"public_addr":"172.21.15.194:6817/367304498","cluster_addr":"172.21.15.194:6819/367304498","heartbeat_back_addr":"172.21.15.194:6823/367304498","heartbeat_front_addr":"172.21.15.194:6821/367304498","state":["exists","up"]},{"osd":10,"uuid":"76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6825","nonce":3349503344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6827","nonce":3349503344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6831","nonce":3349503344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6829","nonce":3349503344}]},"public_addr":"172.21.15.194:6825/3349503344","cluster_addr":"172.21.15.194:6827/3349503344","heartbeat_back_addr":"172.21.15.194:6831/3349503344","heartbeat_front_addr":"172.21.15.194:6829/3349503344","state":["exists","up"]},{"osd":11,"uuid":"f71641eb-1331-4b39-9c0e-bf98e91ed295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6809","nonce":2531896090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6811","nonce":2531896090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6815","nonce":2531896090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6813","nonce":2531896090}]},"public_addr":"172.21.15.194:6809/2531896090","cluster_addr":"172.21.15.194:6811/2531896090","heartbeat_back_addr":"172.21.15.194:6815/2531896090","heartbeat_front_addr":"172.21.15.194:6813/2531896090","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:45.852767+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:45.938876+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.592432+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.591765+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.097182+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.295219+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.146281+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:07:48.985 DEBUG:teuthology.misc:12 of 12 OSDs are up 2024-09-16T04:07:48.985 INFO:tasks.ceph:Creating RBD pool 2024-09-16T04:07:48.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T04:07:49.929 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'rbd' created 2024-09-16T04:07:49.945 DEBUG:teuthology.orchestra.run.smithi016:> rbd --cluster ceph pool init rbd 2024-09-16T04:07:52.952 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T04:07:52.953 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T04:07:52.953 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T04:07:52.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T04:07:52.953 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T04:07:53.314 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:07:53.325 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":159397484},{"type":"v1","addr":"172.21.15.174:6801","nonce":159397484}]},"active_addr":"172.21.15.174:6801/159397484","active_change":"2024-09-16T04:07:40.663135+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":654099619}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":2703591677}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":1019903265}]}]} 2024-09-16T04:07:53.327 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T04:07:53.329 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T04:07:53.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:07:53.628 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:07:53.629 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"44381953-4509-4b15-853e-3887928dfa29","created":"2024-09-16T04:07:36.650572+0000","modified":"2024-09-16T04:07:52.935596+0000","last_up_change":"2024-09-16T04:07:48.919317+0000","last_in_change":"2024-09-16T04:07:39.793439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:07:48.723300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:07:49.309634+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":0.75,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4506422f-7ccf-46d7-af66-62e8cace167a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6825","nonce":3110165165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6827","nonce":3110165165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6831","nonce":3110165165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6829","nonce":3110165165}]},"public_addr":"172.21.15.16:6825/3110165165","cluster_addr":"172.21.15.16:6827/3110165165","heartbeat_back_addr":"172.21.15.16:6831/3110165165","heartbeat_front_addr":"172.21.15.16:6829/3110165165","state":["exists","up"]},{"osd":1,"uuid":"8dc0488e-efb6-4664-a799-5e14c56769b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6810","nonce":3040423711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6814","nonce":3040423711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6822","nonce":3040423711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6818","nonce":3040423711}]},"public_addr":"172.21.15.16:6810/3040423711","cluster_addr":"172.21.15.16:6814/3040423711","heartbeat_back_addr":"172.21.15.16:6822/3040423711","heartbeat_front_addr":"172.21.15.16:6818/3040423711","state":["exists","up"]},{"osd":2,"uuid":"1686e657-c1f3-499b-8b10-67f428c5c216","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6801","nonce":2266032822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6803","nonce":2266032822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6807","nonce":2266032822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6805","nonce":2266032822}]},"public_addr":"172.21.15.16:6801/2266032822","cluster_addr":"172.21.15.16:6803/2266032822","heartbeat_back_addr":"172.21.15.16:6807/2266032822","heartbeat_front_addr":"172.21.15.16:6805/2266032822","state":["exists","up"]},{"osd":3,"uuid":"730ad37c-1f5f-4d23-a1a7-c75320506665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6811","nonce":3389562884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6815","nonce":3389562884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6823","nonce":3389562884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6819","nonce":3389562884}]},"public_addr":"172.21.15.16:6811/3389562884","cluster_addr":"172.21.15.16:6815/3389562884","heartbeat_back_addr":"172.21.15.16:6823/3389562884","heartbeat_front_addr":"172.21.15.16:6819/3389562884","state":["exists","up"]},{"osd":4,"uuid":"d82c5c44-72fe-4680-9c04-ab4e0775b929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6811","nonce":1733996396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6813","nonce":1733996396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6817","nonce":1733996396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6815","nonce":1733996396}]},"public_addr":"172.21.15.174:6811/1733996396","cluster_addr":"172.21.15.174:6813/1733996396","heartbeat_back_addr":"172.21.15.174:6817/1733996396","heartbeat_front_addr":"172.21.15.174:6815/1733996396","state":["exists","up"]},{"osd":5,"uuid":"41502441-b0d9-460f-92c0-b8b3a915cac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6827","nonce":2956753716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6829","nonce":2956753716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6833","nonce":2956753716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6831","nonce":2956753716}]},"public_addr":"172.21.15.174:6827/2956753716","cluster_addr":"172.21.15.174:6829/2956753716","heartbeat_back_addr":"172.21.15.174:6833/2956753716","heartbeat_front_addr":"172.21.15.174:6831/2956753716","state":["exists","up"]},{"osd":6,"uuid":"45cc7521-11db-4496-8d0c-1abc42962810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6803","nonce":2727832023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6805","nonce":2727832023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6809","nonce":2727832023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6807","nonce":2727832023}]},"public_addr":"172.21.15.174:6803/2727832023","cluster_addr":"172.21.15.174:6805/2727832023","heartbeat_back_addr":"172.21.15.174:6809/2727832023","heartbeat_front_addr":"172.21.15.174:6807/2727832023","state":["exists","up"]},{"osd":7,"uuid":"1b5d6337-73c3-494b-91cd-fcfa7d307050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6819","nonce":2338392542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6821","nonce":2338392542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6825","nonce":2338392542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6823","nonce":2338392542}]},"public_addr":"172.21.15.174:6819/2338392542","cluster_addr":"172.21.15.174:6821/2338392542","heartbeat_back_addr":"172.21.15.174:6825/2338392542","heartbeat_front_addr":"172.21.15.174:6823/2338392542","state":["exists","up"]},{"osd":8,"uuid":"766efffa-3aff-4aa3-9cf8-88c5c7986d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621487148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621487148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621487148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621487148}]},"public_addr":"172.21.15.194:6801/2621487148","cluster_addr":"172.21.15.194:6803/2621487148","heartbeat_back_addr":"172.21.15.194:6807/2621487148","heartbeat_front_addr":"172.21.15.194:6805/2621487148","state":["exists","up"]},{"osd":9,"uuid":"755cc9aa-2f6f-4d7a-84cf-7feb74d65c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6817","nonce":367304498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6819","nonce":367304498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6823","nonce":367304498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6821","nonce":367304498}]},"public_addr":"172.21.15.194:6817/367304498","cluster_addr":"172.21.15.194:6819/367304498","heartbeat_back_addr":"172.21.15.194:6823/367304498","heartbeat_front_addr":"172.21.15.194:6821/367304498","state":["exists","up"]},{"osd":10,"uuid":"76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6825","nonce":3349503344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6827","nonce":3349503344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6831","nonce":3349503344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6829","nonce":3349503344}]},"public_addr":"172.21.15.194:6825/3349503344","cluster_addr":"172.21.15.194:6827/3349503344","heartbeat_back_addr":"172.21.15.194:6831/3349503344","heartbeat_front_addr":"172.21.15.194:6829/3349503344","state":["exists","up"]},{"osd":11,"uuid":"f71641eb-1331-4b39-9c0e-bf98e91ed295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6809","nonce":2531896090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6811","nonce":2531896090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6815","nonce":2531896090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6813","nonce":2531896090}]},"public_addr":"172.21.15.194:6809/2531896090","cluster_addr":"172.21.15.194:6811/2531896090","heartbeat_back_addr":"172.21.15.194:6815/2531896090","heartbeat_front_addr":"172.21.15.194:6813/2531896090","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:45.852767+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:45.938876+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.175090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:45.893958+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.162300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:47.123532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.160601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.592432+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.591765+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.097182+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.295219+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.146281+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:07:53.641 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T04:07:53.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:07:53.939 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:07:53.939 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"44381953-4509-4b15-853e-3887928dfa29","created":"2024-09-16T04:07:36.650572+0000","modified":"2024-09-16T04:07:52.935596+0000","last_up_change":"2024-09-16T04:07:48.919317+0000","last_in_change":"2024-09-16T04:07:39.793439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:07:48.723300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:07:49.309634+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":0.75,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4506422f-7ccf-46d7-af66-62e8cace167a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6825","nonce":3110165165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6827","nonce":3110165165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6831","nonce":3110165165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6829","nonce":3110165165}]},"public_addr":"172.21.15.16:6825/3110165165","cluster_addr":"172.21.15.16:6827/3110165165","heartbeat_back_addr":"172.21.15.16:6831/3110165165","heartbeat_front_addr":"172.21.15.16:6829/3110165165","state":["exists","up"]},{"osd":1,"uuid":"8dc0488e-efb6-4664-a799-5e14c56769b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6810","nonce":3040423711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6814","nonce":3040423711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6822","nonce":3040423711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3040423711},{"type":"v1","addr":"172.21.15.16:6818","nonce":3040423711}]},"public_addr":"172.21.15.16:6810/3040423711","cluster_addr":"172.21.15.16:6814/3040423711","heartbeat_back_addr":"172.21.15.16:6822/3040423711","heartbeat_front_addr":"172.21.15.16:6818/3040423711","state":["exists","up"]},{"osd":2,"uuid":"1686e657-c1f3-499b-8b10-67f428c5c216","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6801","nonce":2266032822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6803","nonce":2266032822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6807","nonce":2266032822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6805","nonce":2266032822}]},"public_addr":"172.21.15.16:6801/2266032822","cluster_addr":"172.21.15.16:6803/2266032822","heartbeat_back_addr":"172.21.15.16:6807/2266032822","heartbeat_front_addr":"172.21.15.16:6805/2266032822","state":["exists","up"]},{"osd":3,"uuid":"730ad37c-1f5f-4d23-a1a7-c75320506665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6811","nonce":3389562884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6815","nonce":3389562884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6823","nonce":3389562884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6819","nonce":3389562884}]},"public_addr":"172.21.15.16:6811/3389562884","cluster_addr":"172.21.15.16:6815/3389562884","heartbeat_back_addr":"172.21.15.16:6823/3389562884","heartbeat_front_addr":"172.21.15.16:6819/3389562884","state":["exists","up"]},{"osd":4,"uuid":"d82c5c44-72fe-4680-9c04-ab4e0775b929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6811","nonce":1733996396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6813","nonce":1733996396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6817","nonce":1733996396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6815","nonce":1733996396}]},"public_addr":"172.21.15.174:6811/1733996396","cluster_addr":"172.21.15.174:6813/1733996396","heartbeat_back_addr":"172.21.15.174:6817/1733996396","heartbeat_front_addr":"172.21.15.174:6815/1733996396","state":["exists","up"]},{"osd":5,"uuid":"41502441-b0d9-460f-92c0-b8b3a915cac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6827","nonce":2956753716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6829","nonce":2956753716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6833","nonce":2956753716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6831","nonce":2956753716}]},"public_addr":"172.21.15.174:6827/2956753716","cluster_addr":"172.21.15.174:6829/2956753716","heartbeat_back_addr":"172.21.15.174:6833/2956753716","heartbeat_front_addr":"172.21.15.174:6831/2956753716","state":["exists","up"]},{"osd":6,"uuid":"45cc7521-11db-4496-8d0c-1abc42962810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6803","nonce":2727832023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6805","nonce":2727832023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6809","nonce":2727832023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6807","nonce":2727832023}]},"public_addr":"172.21.15.174:6803/2727832023","cluster_addr":"172.21.15.174:6805/2727832023","heartbeat_back_addr":"172.21.15.174:6809/2727832023","heartbeat_front_addr":"172.21.15.174:6807/2727832023","state":["exists","up"]},{"osd":7,"uuid":"1b5d6337-73c3-494b-91cd-fcfa7d307050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6819","nonce":2338392542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6821","nonce":2338392542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6825","nonce":2338392542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6823","nonce":2338392542}]},"public_addr":"172.21.15.174:6819/2338392542","cluster_addr":"172.21.15.174:6821/2338392542","heartbeat_back_addr":"172.21.15.174:6825/2338392542","heartbeat_front_addr":"172.21.15.174:6823/2338392542","state":["exists","up"]},{"osd":8,"uuid":"766efffa-3aff-4aa3-9cf8-88c5c7986d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621487148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621487148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621487148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621487148}]},"public_addr":"172.21.15.194:6801/2621487148","cluster_addr":"172.21.15.194:6803/2621487148","heartbeat_back_addr":"172.21.15.194:6807/2621487148","heartbeat_front_addr":"172.21.15.194:6805/2621487148","state":["exists","up"]},{"osd":9,"uuid":"755cc9aa-2f6f-4d7a-84cf-7feb74d65c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6817","nonce":367304498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6819","nonce":367304498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6823","nonce":367304498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6821","nonce":367304498}]},"public_addr":"172.21.15.194:6817/367304498","cluster_addr":"172.21.15.194:6819/367304498","heartbeat_back_addr":"172.21.15.194:6823/367304498","heartbeat_front_addr":"172.21.15.194:6821/367304498","state":["exists","up"]},{"osd":10,"uuid":"76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6825","nonce":3349503344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6827","nonce":3349503344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6831","nonce":3349503344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6829","nonce":3349503344}]},"public_addr":"172.21.15.194:6825/3349503344","cluster_addr":"172.21.15.194:6827/3349503344","heartbeat_back_addr":"172.21.15.194:6831/3349503344","heartbeat_front_addr":"172.21.15.194:6829/3349503344","state":["exists","up"]},{"osd":11,"uuid":"f71641eb-1331-4b39-9c0e-bf98e91ed295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6809","nonce":2531896090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6811","nonce":2531896090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6815","nonce":2531896090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6813","nonce":2531896090}]},"public_addr":"172.21.15.194:6809/2531896090","cluster_addr":"172.21.15.194:6811/2531896090","heartbeat_back_addr":"172.21.15.194:6815/2531896090","heartbeat_front_addr":"172.21.15.194:6813/2531896090","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:45.852767+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:45.938876+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.175090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:45.893958+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.162300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:47.123532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.160601+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.592432+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.591765+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.097182+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.295219+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.146281+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:07:53.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:07:53.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:07:53.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:07:53.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:07:53.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T04:07:53.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T04:07:53.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T04:07:53.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T04:07:53.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T04:07:53.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T04:07:53.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T04:07:53.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T04:07:54.259 INFO:teuthology.orchestra.run.smithi016.stdout:81604378629 2024-09-16T04:07:54.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:07:54.294 INFO:teuthology.orchestra.run.smithi016.stdout:81604378627 2024-09-16T04:07:54.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:07:54.297 INFO:teuthology.orchestra.run.smithi016.stdout:77309411331 2024-09-16T04:07:54.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:07:54.298 INFO:teuthology.orchestra.run.smithi016.stdout:77309411331 2024-09-16T04:07:54.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:07:54.305 INFO:teuthology.orchestra.run.smithi016.stdout:81604378626 2024-09-16T04:07:54.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:07:54.311 INFO:teuthology.orchestra.run.smithi016.stdout:73014444036 2024-09-16T04:07:54.311 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:07:54.318 INFO:teuthology.orchestra.run.smithi016.stdout:77309411333 2024-09-16T04:07:54.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:07:54.328 INFO:teuthology.orchestra.run.smithi016.stdout:81604378626 2024-09-16T04:07:54.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:07:54.330 INFO:teuthology.orchestra.run.smithi016.stdout:77309411331 2024-09-16T04:07:54.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:07:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:77309411333 2024-09-16T04:07:54.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:07:54.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:07:54.402+0000 7fc379d9b640 0 --2- 172.21.15.16:0/224678429 >> v2:172.21.15.16:3300/0 conn(0x7fc37c0b4250 0x7fc37c0b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:07:54.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:07:54.421+0000 7f3c237fe640 0 --2- 172.21.15.16:0/3393607029 >> v2:172.21.15.16:3300/0 conn(0x7f3c240b36c0 0x7f3c240ad490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:07:54.556 INFO:teuthology.orchestra.run.smithi016.stdout:81604378627 2024-09-16T04:07:54.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:07:54.572 INFO:teuthology.orchestra.run.smithi016.stdout:77309411332 2024-09-16T04:07:54.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:07:54.999 INFO:teuthology.orchestra.run.smithi016.stdout:81604378628 2024-09-16T04:07:55.038 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378629 got 81604378628 for osd.3 2024-09-16T04:07:55.052 INFO:teuthology.orchestra.run.smithi016.stdout:77309411331 2024-09-16T04:07:55.078 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411331 got 77309411331 for osd.11 2024-09-16T04:07:55.078 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.091 INFO:teuthology.orchestra.run.smithi016.stdout:77309411331 2024-09-16T04:07:55.125 INFO:teuthology.orchestra.run.smithi016.stdout:73014444036 2024-09-16T04:07:55.125 INFO:teuthology.orchestra.run.smithi016.stdout:81604378627 2024-09-16T04:07:55.125 INFO:teuthology.orchestra.run.smithi016.stdout:81604378626 2024-09-16T04:07:55.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411331 got 77309411331 for osd.9 2024-09-16T04:07:55.138 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.144 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378627 got 81604378627 for osd.6 2024-09-16T04:07:55.145 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.147 INFO:teuthology.orchestra.run.smithi016.stdout:77309411333 2024-09-16T04:07:55.154 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378626 got 81604378626 for osd.5 2024-09-16T04:07:55.155 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.159 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.0 2024-09-16T04:07:55.160 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.161 INFO:teuthology.orchestra.run.smithi016.stdout:77309411333 2024-09-16T04:07:55.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411333 got 77309411333 for osd.10 2024-09-16T04:07:55.168 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.180 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411333 got 77309411333 for osd.7 2024-09-16T04:07:55.181 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.188 INFO:teuthology.orchestra.run.smithi016.stdout:81604378626 2024-09-16T04:07:55.202 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378626 got 81604378626 for osd.4 2024-09-16T04:07:55.202 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.204 INFO:teuthology.orchestra.run.smithi016.stdout:77309411331 2024-09-16T04:07:55.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411331 got 77309411331 for osd.8 2024-09-16T04:07:55.218 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi016.stdout:81604378627 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi016.stdout:77309411332 2024-09-16T04:07:55.305 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378627 got 81604378627 for osd.2 2024-09-16T04:07:55.305 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:55.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411332 got 77309411332 for osd.1 2024-09-16T04:07:55.314 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:56.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:07:56.343 INFO:teuthology.orchestra.run.smithi016.stdout:81604378629 2024-09-16T04:07:56.356 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378629 got 81604378629 for osd.3 2024-09-16T04:07:56.356 DEBUG:teuthology.parallel:result is None 2024-09-16T04:07:56.356 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:07:56.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:07:56.648 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:07:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:07:56.660 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-16T04:07:55.425042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":340764,"kb_used_data":3640,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1132121316,"statfs":{"total":1159641169920,"available":1159292227584,"internally_reserved":0,"allocated":3727360,"data_stored":2076970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":344634684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.756134"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.941363+0000","last_change":"2024-09-16T04:07:52.941363+0000","last_active":"2024-09-16T04:07:52.941363+0000","last_peered":"2024-09-16T04:07:52.941363+0000","last_clean":"2024-09-16T04:07:52.941363+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:07:52.941363+0000","last_undegraded":"2024-09-16T04:07:52.941363+0000","last_fullsized":"2024-09-16T04:07:52.941363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:59.932581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.946321+0000","last_change":"2024-09-16T04:07:52.946321+0000","last_active":"2024-09-16T04:07:52.946321+0000","last_peered":"2024-09-16T04:07:52.946321+0000","last_clean":"2024-09-16T04:07:52.946321+0000","last_became_active":"2024-09-16T04:07:50.937245+0000","last_became_peered":"2024-09-16T04:07:50.937245+0000","last_unstale":"2024-09-16T04:07:52.946321+0000","last_undegraded":"2024-09-16T04:07:52.946321+0000","last_fullsized":"2024-09-16T04:07:52.946321+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:18.129176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051620099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.939889+0000","last_change":"2024-09-16T04:07:52.939889+0000","last_active":"2024-09-16T04:07:52.939889+0000","last_peered":"2024-09-16T04:07:52.939889+0000","last_clean":"2024-09-16T04:07:52.939889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:07:52.939889+0000","last_undegraded":"2024-09-16T04:07:52.939889+0000","last_fullsized":"2024-09-16T04:07:52.939889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:16.774685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.946320+0000","last_change":"2024-09-16T04:07:52.946320+0000","last_active":"2024-09-16T04:07:52.946320+0000","last_peered":"2024-09-16T04:07:52.946320+0000","last_clean":"2024-09-16T04:07:52.946320+0000","last_became_active":"2024-09-16T04:07:50.936587+0000","last_became_peered":"2024-09-16T04:07:50.936587+0000","last_unstale":"2024-09-16T04:07:52.946320+0000","last_undegraded":"2024-09-16T04:07:52.946320+0000","last_fullsized":"2024-09-16T04:07:52.946320+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:13.247770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051151899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"23'2","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.945414+0000","last_change":"2024-09-16T04:07:52.944002+0000","last_active":"2024-09-16T04:07:52.945414+0000","last_peered":"2024-09-16T04:07:52.945414+0000","last_clean":"2024-09-16T04:07:52.945414+0000","last_became_active":"2024-09-16T04:07:50.935310+0000","last_became_peered":"2024-09-16T04:07:50.935310+0000","last_unstale":"2024-09-16T04:07:52.945414+0000","last_undegraded":"2024-09-16T04:07:52.945414+0000","last_fullsized":"2024-09-16T04:07:52.945414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:03.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:53.018517+0000","last_change":"2024-09-16T04:07:53.018517+0000","last_active":"2024-09-16T04:07:53.018517+0000","last_peered":"2024-09-16T04:07:53.018517+0000","last_clean":"2024-09-16T04:07:53.018517+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:07:53.018517+0000","last_undegraded":"2024-09-16T04:07:53.018517+0000","last_fullsized":"2024-09-16T04:07:53.018517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.942253+0000","last_change":"2024-09-16T04:07:52.942253+0000","last_active":"2024-09-16T04:07:52.942253+0000","last_peered":"2024-09-16T04:07:52.942253+0000","last_clean":"2024-09-16T04:07:52.942253+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:07:52.942253+0000","last_undegraded":"2024-09-16T04:07:52.942253+0000","last_fullsized":"2024-09-16T04:07:52.942253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.104501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"21'1","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.943663+0000","last_change":"2024-09-16T04:07:52.943663+0000","last_active":"2024-09-16T04:07:52.943663+0000","last_peered":"2024-09-16T04:07:52.943663+0000","last_clean":"2024-09-16T04:07:52.943663+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:07:52.943663+0000","last_undegraded":"2024-09-16T04:07:52.943663+0000","last_fullsized":"2024-09-16T04:07:52.943663+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.855553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"23'137","reported_seq":189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:53.393583+0000","last_change":"2024-09-16T04:07:50.119798+0000","last_active":"2024-09-16T04:07:53.393583+0000","last_peered":"2024-09-16T04:07:53.393583+0000","last_clean":"2024-09-16T04:07:53.393583+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:07:53.393583+0000","last_undegraded":"2024-09-16T04:07:53.393583+0000","last_fullsized":"2024-09-16T04:07:53.393583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:07:48.919317+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:53.937788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378628,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":84942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":753664,"data_stored":613851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411331,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":753664,"data_stored":613851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411332,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":84942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:07:56.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:07:56.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:07:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:07:56.968 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T04:07:56.669246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":340764,"kb_used_data":3640,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1132121316,"statfs":{"total":1159641169920,"available":1159292227584,"internally_reserved":0,"allocated":3727360,"data_stored":2076970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":344634684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000338"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.941363+0000","last_change":"2024-09-16T04:07:52.941363+0000","last_active":"2024-09-16T04:07:52.941363+0000","last_peered":"2024-09-16T04:07:52.941363+0000","last_clean":"2024-09-16T04:07:52.941363+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:07:52.941363+0000","last_undegraded":"2024-09-16T04:07:52.941363+0000","last_fullsized":"2024-09-16T04:07:52.941363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:59.932581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.946321+0000","last_change":"2024-09-16T04:07:52.946321+0000","last_active":"2024-09-16T04:07:52.946321+0000","last_peered":"2024-09-16T04:07:52.946321+0000","last_clean":"2024-09-16T04:07:52.946321+0000","last_became_active":"2024-09-16T04:07:50.937245+0000","last_became_peered":"2024-09-16T04:07:50.937245+0000","last_unstale":"2024-09-16T04:07:52.946321+0000","last_undegraded":"2024-09-16T04:07:52.946321+0000","last_fullsized":"2024-09-16T04:07:52.946321+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:18.129176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051620099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.939889+0000","last_change":"2024-09-16T04:07:52.939889+0000","last_active":"2024-09-16T04:07:52.939889+0000","last_peered":"2024-09-16T04:07:52.939889+0000","last_clean":"2024-09-16T04:07:52.939889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:07:52.939889+0000","last_undegraded":"2024-09-16T04:07:52.939889+0000","last_fullsized":"2024-09-16T04:07:52.939889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:16.774685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.946320+0000","last_change":"2024-09-16T04:07:52.946320+0000","last_active":"2024-09-16T04:07:52.946320+0000","last_peered":"2024-09-16T04:07:52.946320+0000","last_clean":"2024-09-16T04:07:52.946320+0000","last_became_active":"2024-09-16T04:07:50.936587+0000","last_became_peered":"2024-09-16T04:07:50.936587+0000","last_unstale":"2024-09-16T04:07:52.946320+0000","last_undegraded":"2024-09-16T04:07:52.946320+0000","last_fullsized":"2024-09-16T04:07:52.946320+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:13.247770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051151899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"23'2","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.945414+0000","last_change":"2024-09-16T04:07:52.944002+0000","last_active":"2024-09-16T04:07:52.945414+0000","last_peered":"2024-09-16T04:07:52.945414+0000","last_clean":"2024-09-16T04:07:52.945414+0000","last_became_active":"2024-09-16T04:07:50.935310+0000","last_became_peered":"2024-09-16T04:07:50.935310+0000","last_unstale":"2024-09-16T04:07:52.945414+0000","last_undegraded":"2024-09-16T04:07:52.945414+0000","last_fullsized":"2024-09-16T04:07:52.945414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:03.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:53.018517+0000","last_change":"2024-09-16T04:07:53.018517+0000","last_active":"2024-09-16T04:07:53.018517+0000","last_peered":"2024-09-16T04:07:53.018517+0000","last_clean":"2024-09-16T04:07:53.018517+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:07:53.018517+0000","last_undegraded":"2024-09-16T04:07:53.018517+0000","last_fullsized":"2024-09-16T04:07:53.018517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.942253+0000","last_change":"2024-09-16T04:07:52.942253+0000","last_active":"2024-09-16T04:07:52.942253+0000","last_peered":"2024-09-16T04:07:52.942253+0000","last_clean":"2024-09-16T04:07:52.942253+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:07:52.942253+0000","last_undegraded":"2024-09-16T04:07:52.942253+0000","last_fullsized":"2024-09-16T04:07:52.942253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.104501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"21'1","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.943663+0000","last_change":"2024-09-16T04:07:52.943663+0000","last_active":"2024-09-16T04:07:52.943663+0000","last_peered":"2024-09-16T04:07:52.943663+0000","last_clean":"2024-09-16T04:07:52.943663+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:07:52.943663+0000","last_undegraded":"2024-09-16T04:07:52.943663+0000","last_fullsized":"2024-09-16T04:07:52.943663+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.855553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"23'137","reported_seq":189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:53.393583+0000","last_change":"2024-09-16T04:07:50.119798+0000","last_active":"2024-09-16T04:07:53.393583+0000","last_peered":"2024-09-16T04:07:53.393583+0000","last_clean":"2024-09-16T04:07:53.393583+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:07:53.393583+0000","last_undegraded":"2024-09-16T04:07:53.393583+0000","last_fullsized":"2024-09-16T04:07:53.393583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:07:48.919317+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:53.937788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378628,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":84942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":753664,"data_stored":613851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411331,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":753664,"data_stored":613851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411332,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":84942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:07:56.969 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:07:56.969 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T04:07:56.969 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T04:07:56.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T04:07:57.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:07:57.302 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T04:07:57.315 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T04:07:57.316 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T04:07:57.331 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T04:07:57.331 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}} 2024-09-16T04:07:57.331 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-16T04:07:57.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-16T04:07:57.645 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T04:07:57.646 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:07:57.646 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:07:57.936 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:07:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:07:57.948 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T04:07:56.669246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":340764,"kb_used_data":3640,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1132121316,"statfs":{"total":1159641169920,"available":1159292227584,"internally_reserved":0,"allocated":3727360,"data_stored":2076970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":344634684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000338"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.941363+0000","last_change":"2024-09-16T04:07:52.941363+0000","last_active":"2024-09-16T04:07:52.941363+0000","last_peered":"2024-09-16T04:07:52.941363+0000","last_clean":"2024-09-16T04:07:52.941363+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:07:52.941363+0000","last_undegraded":"2024-09-16T04:07:52.941363+0000","last_fullsized":"2024-09-16T04:07:52.941363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:59.932581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.946321+0000","last_change":"2024-09-16T04:07:52.946321+0000","last_active":"2024-09-16T04:07:52.946321+0000","last_peered":"2024-09-16T04:07:52.946321+0000","last_clean":"2024-09-16T04:07:52.946321+0000","last_became_active":"2024-09-16T04:07:50.937245+0000","last_became_peered":"2024-09-16T04:07:50.937245+0000","last_unstale":"2024-09-16T04:07:52.946321+0000","last_undegraded":"2024-09-16T04:07:52.946321+0000","last_fullsized":"2024-09-16T04:07:52.946321+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:18.129176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051620099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.939889+0000","last_change":"2024-09-16T04:07:52.939889+0000","last_active":"2024-09-16T04:07:52.939889+0000","last_peered":"2024-09-16T04:07:52.939889+0000","last_clean":"2024-09-16T04:07:52.939889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:07:52.939889+0000","last_undegraded":"2024-09-16T04:07:52.939889+0000","last_fullsized":"2024-09-16T04:07:52.939889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:16.774685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.946320+0000","last_change":"2024-09-16T04:07:52.946320+0000","last_active":"2024-09-16T04:07:52.946320+0000","last_peered":"2024-09-16T04:07:52.946320+0000","last_clean":"2024-09-16T04:07:52.946320+0000","last_became_active":"2024-09-16T04:07:50.936587+0000","last_became_peered":"2024-09-16T04:07:50.936587+0000","last_unstale":"2024-09-16T04:07:52.946320+0000","last_undegraded":"2024-09-16T04:07:52.946320+0000","last_fullsized":"2024-09-16T04:07:52.946320+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:13.247770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051151899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"23'2","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.945414+0000","last_change":"2024-09-16T04:07:52.944002+0000","last_active":"2024-09-16T04:07:52.945414+0000","last_peered":"2024-09-16T04:07:52.945414+0000","last_clean":"2024-09-16T04:07:52.945414+0000","last_became_active":"2024-09-16T04:07:50.935310+0000","last_became_peered":"2024-09-16T04:07:50.935310+0000","last_unstale":"2024-09-16T04:07:52.945414+0000","last_undegraded":"2024-09-16T04:07:52.945414+0000","last_fullsized":"2024-09-16T04:07:52.945414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:03.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:53.018517+0000","last_change":"2024-09-16T04:07:53.018517+0000","last_active":"2024-09-16T04:07:53.018517+0000","last_peered":"2024-09-16T04:07:53.018517+0000","last_clean":"2024-09-16T04:07:53.018517+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:07:53.018517+0000","last_undegraded":"2024-09-16T04:07:53.018517+0000","last_fullsized":"2024-09-16T04:07:53.018517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.942253+0000","last_change":"2024-09-16T04:07:52.942253+0000","last_active":"2024-09-16T04:07:52.942253+0000","last_peered":"2024-09-16T04:07:52.942253+0000","last_clean":"2024-09-16T04:07:52.942253+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:07:52.942253+0000","last_undegraded":"2024-09-16T04:07:52.942253+0000","last_fullsized":"2024-09-16T04:07:52.942253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.104501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"21'1","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.943663+0000","last_change":"2024-09-16T04:07:52.943663+0000","last_active":"2024-09-16T04:07:52.943663+0000","last_peered":"2024-09-16T04:07:52.943663+0000","last_clean":"2024-09-16T04:07:52.943663+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:07:52.943663+0000","last_undegraded":"2024-09-16T04:07:52.943663+0000","last_fullsized":"2024-09-16T04:07:52.943663+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.855553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"23'137","reported_seq":189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:53.393583+0000","last_change":"2024-09-16T04:07:50.119798+0000","last_active":"2024-09-16T04:07:53.393583+0000","last_peered":"2024-09-16T04:07:53.393583+0000","last_clean":"2024-09-16T04:07:53.393583+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:07:53.393583+0000","last_undegraded":"2024-09-16T04:07:53.393583+0000","last_fullsized":"2024-09-16T04:07:53.393583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:07:48.919317+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:53.937788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378628,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":84942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":753664,"data_stored":613851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411331,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":753664,"data_stored":613851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411332,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":84942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:07:57.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:07:58.246 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:07:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:07:58.259 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T04:07:56.669246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":340764,"kb_used_data":3640,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1132121316,"statfs":{"total":1159641169920,"available":1159292227584,"internally_reserved":0,"allocated":3727360,"data_stored":2076970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":344634684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000338"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.941363+0000","last_change":"2024-09-16T04:07:52.941363+0000","last_active":"2024-09-16T04:07:52.941363+0000","last_peered":"2024-09-16T04:07:52.941363+0000","last_clean":"2024-09-16T04:07:52.941363+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:07:52.941363+0000","last_undegraded":"2024-09-16T04:07:52.941363+0000","last_fullsized":"2024-09-16T04:07:52.941363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:59.932581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.946321+0000","last_change":"2024-09-16T04:07:52.946321+0000","last_active":"2024-09-16T04:07:52.946321+0000","last_peered":"2024-09-16T04:07:52.946321+0000","last_clean":"2024-09-16T04:07:52.946321+0000","last_became_active":"2024-09-16T04:07:50.937245+0000","last_became_peered":"2024-09-16T04:07:50.937245+0000","last_unstale":"2024-09-16T04:07:52.946321+0000","last_undegraded":"2024-09-16T04:07:52.946321+0000","last_fullsized":"2024-09-16T04:07:52.946321+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:18.129176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051620099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.939889+0000","last_change":"2024-09-16T04:07:52.939889+0000","last_active":"2024-09-16T04:07:52.939889+0000","last_peered":"2024-09-16T04:07:52.939889+0000","last_clean":"2024-09-16T04:07:52.939889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:07:52.939889+0000","last_undegraded":"2024-09-16T04:07:52.939889+0000","last_fullsized":"2024-09-16T04:07:52.939889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:16.774685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.946320+0000","last_change":"2024-09-16T04:07:52.946320+0000","last_active":"2024-09-16T04:07:52.946320+0000","last_peered":"2024-09-16T04:07:52.946320+0000","last_clean":"2024-09-16T04:07:52.946320+0000","last_became_active":"2024-09-16T04:07:50.936587+0000","last_became_peered":"2024-09-16T04:07:50.936587+0000","last_unstale":"2024-09-16T04:07:52.946320+0000","last_undegraded":"2024-09-16T04:07:52.946320+0000","last_fullsized":"2024-09-16T04:07:52.946320+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:13.247770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051151899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"23'2","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.945414+0000","last_change":"2024-09-16T04:07:52.944002+0000","last_active":"2024-09-16T04:07:52.945414+0000","last_peered":"2024-09-16T04:07:52.945414+0000","last_clean":"2024-09-16T04:07:52.945414+0000","last_became_active":"2024-09-16T04:07:50.935310+0000","last_became_peered":"2024-09-16T04:07:50.935310+0000","last_unstale":"2024-09-16T04:07:52.945414+0000","last_undegraded":"2024-09-16T04:07:52.945414+0000","last_fullsized":"2024-09-16T04:07:52.945414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:03.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:53.018517+0000","last_change":"2024-09-16T04:07:53.018517+0000","last_active":"2024-09-16T04:07:53.018517+0000","last_peered":"2024-09-16T04:07:53.018517+0000","last_clean":"2024-09-16T04:07:53.018517+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:07:53.018517+0000","last_undegraded":"2024-09-16T04:07:53.018517+0000","last_fullsized":"2024-09-16T04:07:53.018517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.942253+0000","last_change":"2024-09-16T04:07:52.942253+0000","last_active":"2024-09-16T04:07:52.942253+0000","last_peered":"2024-09-16T04:07:52.942253+0000","last_clean":"2024-09-16T04:07:52.942253+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:07:52.942253+0000","last_undegraded":"2024-09-16T04:07:52.942253+0000","last_fullsized":"2024-09-16T04:07:52.942253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.104501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"21'1","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:52.943663+0000","last_change":"2024-09-16T04:07:52.943663+0000","last_active":"2024-09-16T04:07:52.943663+0000","last_peered":"2024-09-16T04:07:52.943663+0000","last_clean":"2024-09-16T04:07:52.943663+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:07:52.943663+0000","last_undegraded":"2024-09-16T04:07:52.943663+0000","last_fullsized":"2024-09-16T04:07:52.943663+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.855553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"23'137","reported_seq":189,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-16T04:07:53.393583+0000","last_change":"2024-09-16T04:07:50.119798+0000","last_active":"2024-09-16T04:07:53.393583+0000","last_peered":"2024-09-16T04:07:53.393583+0000","last_clean":"2024-09-16T04:07:53.393583+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:07:53.393583+0000","last_undegraded":"2024-09-16T04:07:53.393583+0000","last_fullsized":"2024-09-16T04:07:53.393583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:07:48.919317+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:53.937788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378628,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":84942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":753664,"data_stored":613851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411331,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":753664,"data_stored":613851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411332,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":84942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":84923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:07:58.260 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:07:58.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T04:07:58.558 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 24 2024-09-16T04:07:58.559 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 44381953-4509-4b15-853e-3887928dfa29 2024-09-16T04:07:58.559 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-16T04:07:36.650572+0000 2024-09-16T04:07:58.559 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-09-16T04:07:55.422797+0000 2024-09-16T04:07:58.559 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T04:07:58.559 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 6 2024-09-16T04:07:58.559 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-09-16T04:07:58.560 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-09-16T04:07:58.560 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-09-16T04:07:58.560 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-09-16T04:07:58.560 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client jewel 2024-09-16T04:07:58.561 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release squid 2024-09-16T04:07:58.561 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-09-16T04:07:58.561 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 11.76 2024-09-16T04:07:58.561 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 23 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 3.00 2024-09-16T04:07:58.561 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 12 2024-09-16T04:07:58.562 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6824/3110165165,v1:172.21.15.16:6825/3110165165] [v2:172.21.15.16:6826/3110165165,v1:172.21.15.16:6827/3110165165] exists,up 4506422f-7ccf-46d7-af66-62e8cace167a 2024-09-16T04:07:58.562 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/3040423711,v1:172.21.15.16:6810/3040423711] [v2:172.21.15.16:6812/3040423711,v1:172.21.15.16:6814/3040423711] exists,up 8dc0488e-efb6-4664-a799-5e14c56769b0 2024-09-16T04:07:58.562 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 19 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/2266032822,v1:172.21.15.16:6801/2266032822] [v2:172.21.15.16:6802/2266032822,v1:172.21.15.16:6803/2266032822] exists,up 1686e657-c1f3-499b-8b10-67f428c5c216 2024-09-16T04:07:58.562 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6809/3389562884,v1:172.21.15.16:6811/3389562884] [v2:172.21.15.16:6813/3389562884,v1:172.21.15.16:6815/3389562884] exists,up 730ad37c-1f5f-4d23-a1a7-c75320506665 2024-09-16T04:07:58.562 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6810/1733996396,v1:172.21.15.174:6811/1733996396] [v2:172.21.15.174:6812/1733996396,v1:172.21.15.174:6813/1733996396] exists,up d82c5c44-72fe-4680-9c04-ab4e0775b929 2024-09-16T04:07:58.563 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 19 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6826/2956753716,v1:172.21.15.174:6827/2956753716] [v2:172.21.15.174:6828/2956753716,v1:172.21.15.174:6829/2956753716] exists,up 41502441-b0d9-460f-92c0-b8b3a915cac5 2024-09-16T04:07:58.563 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 up in weight 1 up_from 19 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6802/2727832023,v1:172.21.15.174:6803/2727832023] [v2:172.21.15.174:6804/2727832023,v1:172.21.15.174:6805/2727832023] exists,up 45cc7521-11db-4496-8d0c-1abc42962810 2024-09-16T04:07:58.563 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6818/2338392542,v1:172.21.15.174:6819/2338392542] [v2:172.21.15.174:6820/2338392542,v1:172.21.15.174:6821/2338392542] exists,up 1b5d6337-73c3-494b-91cd-fcfa7d307050 2024-09-16T04:07:58.563 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6800/2621487148,v1:172.21.15.194:6801/2621487148] [v2:172.21.15.194:6802/2621487148,v1:172.21.15.194:6803/2621487148] exists,up 766efffa-3aff-4aa3-9cf8-88c5c7986d21 2024-09-16T04:07:58.563 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6816/367304498,v1:172.21.15.194:6817/367304498] [v2:172.21.15.194:6818/367304498,v1:172.21.15.194:6819/367304498] exists,up 755cc9aa-2f6f-4d7a-84cf-7feb74d65c48 2024-09-16T04:07:58.564 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6824/3349503344,v1:172.21.15.194:6825/3349503344] [v2:172.21.15.194:6826/3349503344,v1:172.21.15.194:6827/3349503344] exists,up 76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377 2024-09-16T04:07:58.564 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6808/2531896090,v1:172.21.15.194:6809/2531896090] [v2:172.21.15.194:6810/2531896090,v1:172.21.15.194:6811/2531896090] exists,up f71641eb-1331-4b39-9c0e-bf98e91ed295 2024-09-16T04:07:58.571 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6824/3110165165,v1:172.21.15.16:6825/3110165165] [v2:172.21.15.16:6826/3110165165,v1:172.21.15.16:6827/3110165165] exists,up 4506422f-7ccf-46d7-af66-62e8cace167a', 'osd.1 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/3040423711,v1:172.21.15.16:6810/3040423711] [v2:172.21.15.16:6812/3040423711,v1:172.21.15.16:6814/3040423711] exists,up 8dc0488e-efb6-4664-a799-5e14c56769b0', 'osd.2 up in weight 1 up_from 19 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/2266032822,v1:172.21.15.16:6801/2266032822] [v2:172.21.15.16:6802/2266032822,v1:172.21.15.16:6803/2266032822] exists,up 1686e657-c1f3-499b-8b10-67f428c5c216', 'osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6809/3389562884,v1:172.21.15.16:6811/3389562884] [v2:172.21.15.16:6813/3389562884,v1:172.21.15.16:6815/3389562884] exists,up 730ad37c-1f5f-4d23-a1a7-c75320506665', 'osd.4 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6810/1733996396,v1:172.21.15.174:6811/1733996396] [v2:172.21.15.174:6812/1733996396,v1:172.21.15.174:6813/1733996396] exists,up d82c5c44-72fe-4680-9c04-ab4e0775b929', 'osd.5 up in weight 1 up_from 19 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6826/2956753716,v1:172.21.15.174:6827/2956753716] [v2:172.21.15.174:6828/2956753716,v1:172.21.15.174:6829/2956753716] exists,up 41502441-b0d9-460f-92c0-b8b3a915cac5', 'osd.6 up in weight 1 up_from 19 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6802/2727832023,v1:172.21.15.174:6803/2727832023] [v2:172.21.15.174:6804/2727832023,v1:172.21.15.174:6805/2727832023] exists,up 45cc7521-11db-4496-8d0c-1abc42962810', 'osd.7 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6818/2338392542,v1:172.21.15.174:6819/2338392542] [v2:172.21.15.174:6820/2338392542,v1:172.21.15.174:6821/2338392542] exists,up 1b5d6337-73c3-494b-91cd-fcfa7d307050', 'osd.8 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6800/2621487148,v1:172.21.15.194:6801/2621487148] [v2:172.21.15.194:6802/2621487148,v1:172.21.15.194:6803/2621487148] exists,up 766efffa-3aff-4aa3-9cf8-88c5c7986d21', 'osd.9 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6816/367304498,v1:172.21.15.194:6817/367304498] [v2:172.21.15.194:6818/367304498,v1:172.21.15.194:6819/367304498] exists,up 755cc9aa-2f6f-4d7a-84cf-7feb74d65c48', 'osd.10 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6824/3349503344,v1:172.21.15.194:6825/3349503344] [v2:172.21.15.194:6826/3349503344,v1:172.21.15.194:6827/3349503344] exists,up 76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377', 'osd.11 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6808/2531896090,v1:172.21.15.194:6809/2531896090] [v2:172.21.15.194:6810/2531896090,v1:172.21.15.194:6811/2531896090] exists,up f71641eb-1331-4b39-9c0e-bf98e91ed295'] 2024-09-16T04:07:58.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mon.a", 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster": "ceph", 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout: "admin_socket_mode": "", 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_client_required": "cephx, none", 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_cluster_required": "cephx", 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_debug": "true", 2024-09-16T04:07:58.674 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T04:07:58.675 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T04:07:58.675 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_service_required": "cephx", 2024-09-16T04:07:58.675 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T04:07:58.675 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_supported": "cephx", 2024-09-16T04:07:58.675 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio": "true", 2024-09-16T04:07:58.675 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T04:07:58.675 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T04:07:58.675 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio_reap_max": "16", 2024-09-16T04:07:58.676 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_async_discard": "false", 2024-09-16T04:07:58.676 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_block_size": "4096", 2024-09-16T04:07:58.676 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_aio": "false", 2024-09-16T04:07:58.676 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T04:07:58.676 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T04:07:58.676 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T04:07:58.676 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_enable_discard": "false", 2024-09-16T04:07:58.676 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_flock_retry": "3", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_inject_crash": "0", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_ioring": "false", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_ioring_hipri": "false", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T04:07:58.677 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_type": "", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_allocator": "hybrid", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_buffered_io": "true", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T04:07:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_replay_recovery": "false", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_sync_write": "false", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T04:07:58.679 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_allocator": "hybrid", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bdev_label_multi": "true", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_create": "true", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_db_create": "false", 2024-09-16T04:07:58.680 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_db_path": "", 2024-09-16T04:07:58.681 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_db_size": "0", 2024-09-16T04:07:58.681 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_path": "", 2024-09-16T04:07:58.681 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T04:07:58.681 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_size": "107374182400", 2024-09-16T04:07:58.681 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_wal_create": "false", 2024-09-16T04:07:58.681 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_wal_path": "", 2024-09-16T04:07:58.681 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs": "true", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_autotune": "true", 2024-09-16T04:07:58.682 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T04:07:58.683 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T04:07:58.683 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T04:07:58.683 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T04:07:58.683 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_size": "0", 2024-09-16T04:07:58.683 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T04:07:58.683 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T04:07:58.683 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T04:07:58.683 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T04:07:58.683 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_type": "2q", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_clone_cow": "true", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_mode": "none", 2024-09-16T04:07:58.684 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T04:07:58.685 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T04:07:58.685 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T04:07:58.685 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_freelist": "false", 2024-09-16T04:07:58.685 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T04:07:58.685 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T04:07:58.685 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T04:07:58.685 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T04:07:58.685 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T04:07:58.685 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T04:07:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T04:07:58.687 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T04:07:58.687 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T04:07:58.687 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T04:07:58.687 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-16T04:07:58.687 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T04:07:58.687 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T04:07:58.687 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T04:07:58.687 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fail_eio": "false", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T04:07:58.688 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T04:07:58.689 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_blob_size": "0", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T04:07:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T04:07:58.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T04:07:58.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T04:07:58.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-16T04:07:58.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-16T04:07:58.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T04:07:58.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-16T04:07:58.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-16T04:07:58.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T04:07:58.691 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_mem": "512", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_tracing": "false", 2024-09-16T04:07:58.692 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T04:07:58.693 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T04:07:58.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_cluster_require_version": "2", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_require_signatures": "false", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_require_version": "2", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_service_require_signatures": "false", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_service_require_version": "2", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_sign_messages": "true", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "chdir": "", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_acl_type": "", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_asio_thread_count": "2", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_cache_mid": "0.750000", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_cache_size": "16384", 2024-09-16T04:07:58.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_caps_release_delay": "5", 2024-09-16T04:07:58.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_check_pool_perm": "true", 2024-09-16T04:07:58.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T04:07:58.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_force_sync_read": "false", 2024-09-16T04:07:58.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_getattr_caps": "false", 2024-09-16T04:07:58.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_inject_features": "", 2024-09-16T04:07:58.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T04:07:58.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T04:07:58.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_die_on_failed_remount": "false", 2024-09-16T04:07:58.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_dirsize_rbytes": "true", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_force_lazyio": "false", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_fs": "", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_inject_release_failure": "false", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_max_inline_size": "4096", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mds_namespace": "", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_metadata": "", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mount_gid": "-1", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mount_timeout": "300", 2024-09-16T04:07:58.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mount_uid": "-1", 2024-09-16T04:07:58.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mountpoint": "/", 2024-09-16T04:07:58.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_notify_timeout": "10", 2024-09-16T04:07:58.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc": "true", 2024-09-16T04:07:58.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T04:07:58.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T04:07:58.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_max_objects": "1000", 2024-09-16T04:07:58.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_size": "209715200", 2024-09-16T04:07:58.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T04:07:58.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_permissions": "true", 2024-09-16T04:07:58.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_quota": "true", 2024-09-16T04:07:58.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_quota_df": "true", 2024-09-16T04:07:58.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_readahead_max_bytes": "0", 2024-09-16T04:07:58.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_readahead_max_periods": "4", 2024-09-16T04:07:58.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_readahead_min": "131072", 2024-09-16T04:07:58.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_reconnect_stale": "false", 2024-09-16T04:07:58.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_shutdown_timeout": "30", 2024-09-16T04:07:58.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_snapdir": ".snap", 2024-09-16T04:07:58.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_tick_interval": "1", 2024-09-16T04:07:58.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_trace": "", 2024-09-16T04:07:58.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T04:07:58.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_use_faked_inos": "false", 2024-09-16T04:07:58.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_use_random_mds": "false", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_graylog": "false", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_graylog_port": "12201", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_monitors": "default=true", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_syslog": "false", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_syslog_level": "info", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "-", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_network": "", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_network_interface": "", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zlib_isal": "false", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zlib_level": "5", 2024-09-16T04:07:58.701 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zstd_level": "1", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_osd_stat_interval": "0", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_seastar_cpu_cores": "", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_seastar_num_threads": "0", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_location": "", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_location_hook": "", 2024-09-16T04:07:58.702 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_location_hook_timeout": "10", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "daemonize": "false", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "daos_pool": "tank", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_alienstore": "0/5", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_asok": "1/5", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_asok_assert_abort": "false", 2024-09-16T04:07:58.703 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T04:07:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_auth": "20/20", 2024-09-16T04:07:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_bdev": "1/3", 2024-09-16T04:07:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_bluefs": "1/5", 2024-09-16T04:07:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_bluestore": "1/5", 2024-09-16T04:07:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_buffer": "0/1", 2024-09-16T04:07:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T04:07:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T04:07:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_cephsqlite": "0/5", 2024-09-16T04:07:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_client": "0/5", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_compressor": "1/5", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_context": "0/1", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_crimson_interrupt": "0/5", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_crush": "1/1", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_crypto": "1/5", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_cyanstore": "0/5", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_dpdk": "1/5", 2024-09-16T04:07:58.705 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_eventtrace": "1/5", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_filer": "0/1", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_filestore": "1/3", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_finisher": "1/1", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_fuse": "1/5", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_javaclient": "1/5", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_journal": "1/3", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_journaler": "0/5", 2024-09-16T04:07:58.706 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_kstore": "1/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_lockdep": "0/1", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mclock": "1/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds": "1/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_balancer": "1/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_locker": "1/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_log": "1/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_migrator": "1/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_quiesce": "3/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_memstore": "1/5", 2024-09-16T04:07:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mgr": "2/5", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mgrc": "1/5", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mon": "20/20", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_monc": "0/10", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_ms": "1/1", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_none": "0/5", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_objclass": "0/5", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_objectcacher": "0/5", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_objecter": "0/1", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_optracker": "0/5", 2024-09-16T04:07:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_osd": "1/5", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_paxos": "20/20", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_perfcounter": "1/5", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_prioritycache": "1/5", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rados": "0/5", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd": "0/5", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd_replay": "0/5", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_refs": "0/0", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_reserver": "1/1", 2024-09-16T04:07:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw": "1/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_access": "1/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_dbstore": "1/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_flight": "1/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_sync": "1/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rocksdb": "4/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore": "0/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_backref": "0/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_cache": "0/5", 2024-09-16T04:07:58.710 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_device": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_epm": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_journal": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_lba": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_odata": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_omap": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_onode": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_t": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_seastore_tm": "0/5", 2024-09-16T04:07:58.711 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_striper": "0/1", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_test": "0/5", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_throttle": "1/1", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_timer": "0/1", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_tp": "0/5", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_trace": "1/5", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "device_failure_prediction_mode": "none", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-16T04:07:58.712 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_graylog": "false", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_journald": "false", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_stderr": "true", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_syslog": "false", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event_tracing": "false", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_http_port": "9926", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_prio_limit": "5", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_sort_metrics": "true", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "exporter_stats_period": "5", 2024-09-16T04:07:58.713 INFO:teuthology.orchestra.run.smithi016.stdout: "fake_statfs_for_testing": "0", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "fatal_signal_handlers": "true", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filer_max_purge_ops": "10", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filer_max_truncate_ops": "128", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_blackhole": "false", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_btrfs_snap": "true", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_caller_concurrency": "10", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T04:07:58.714 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_omap_check": "false", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_verify_split": "false", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_dump_file": "", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fadvise": "true", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fail_eio": "true", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fd_cache_size": "128", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fiemap": "false", 2024-09-16T04:07:58.715 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_inject_stall": "0", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_journal_parallel": "false", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_journal_trailing": "false", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_journal_writeahead": "false", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_kill_at": "0", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T04:07:58.716 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T04:07:58.717 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_merge_threshold": "-10", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_odsync_write": "false", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_omap_backend_path": "", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_op_threads": "2", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_punch_hole": "false", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T04:07:58.718 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_ops": "50", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T04:07:58.719 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_seek_data_hole": "false", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_sloppy_crc": "false", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_splice": "false", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_split_multiple": "2", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_split_rand_factor": "20", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_update_to": "1000", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T04:07:58.720 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_xfs_extsize": "false", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_zfs_snap": "false", 2024-09-16T04:07:58.721 INFO:teuthology.orchestra.run.smithi016.stdout: "fio_dir": "/tmp/fio", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "44381953-4509-4b15-853e-3887928dfa29", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_allow_other": "true", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_big_writes": "true", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_debug": "false", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_default_permissions": "false", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_disable_pagecache": "false", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_max_write": "0", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_multithreaded": "true", 2024-09-16T04:07:58.722 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_require_active_mds": "true", 2024-09-16T04:07:58.723 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_set_user_groups": "true", 2024-09-16T04:07:58.723 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_splice_move": "true", 2024-09-16T04:07:58.723 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_splice_read": "true", 2024-09-16T04:07:58.723 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_splice_write": "true", 2024-09-16T04:07:58.723 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T04:07:58.723 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T04:07:58.723 INFO:teuthology.orchestra.run.smithi016.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T04:07:58.723 INFO:teuthology.orchestra.run.smithi016.stdout: "gss_target_name": "ceph", 2024-09-16T04:07:58.723 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_file": "", 2024-09-16T04:07:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_inject_failure": "0", 2024-09-16T04:07:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_interval": "5", 2024-09-16T04:07:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: "host": "smithi016", 2024-09-16T04:07:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T04:07:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T04:07:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T04:07:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T04:07:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T04:07:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "inject_early_sigterm": "false", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "jaeger_agent_port": "6799", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "jaeger_tracing_enable": "false", 2024-09-16T04:07:58.725 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_aio": "true", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_align_min_size": "65536", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_block_align": "true", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_block_size": "4096", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_dio": "true", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_discard": "false", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_force_aio": "false", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_ignore_corruption": "false", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_max_write_entries": "100", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_replay_from": "0", 2024-09-16T04:07:58.726 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_write_header_frequency": "0", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_zero_on_create": "false", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "journaler_prefetch_periods": "10", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "journaler_prezero_periods": "5", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "journaler_write_head_interval": "15", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "keyfile": "", 2024-09-16T04:07:58.727 INFO:teuthology.orchestra.run.smithi016.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_backend": "rocksdb", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_max_bytes": "67108864", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_max_ops": "512", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_onode_map_size": "1024", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_sync_transaction": "false", 2024-09-16T04:07:58.728 INFO:teuthology.orchestra.run.smithi016.stdout: "librados_thread_count": "2", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "lockdep": "false", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "lockdep_force_backtrace": "false", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "log_coarse_timestamps": "true", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "log_flush_on_exit": "false", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "log_graylog_port": "12201", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "log_max_new": "1000", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "log_max_recent": "10000", 2024-09-16T04:07:58.729 INFO:teuthology.orchestra.run.smithi016.stdout: "log_stderr_prefix": "", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": "true", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_graylog": "false", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_journald": "false", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_stderr": "false", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_syslog": "false", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_action_on_write_error": "1", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_alternate_name_max": "8192", 2024-09-16T04:07:58.730 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_asio_thread_count": "2", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_export_pin": "true", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_interval": "10", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_max": "-1", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_max_until": "-1", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_merge_size": "50", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T04:07:58.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_mode": "0", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_overload_epochs": "2", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_bits": "3", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T04:07:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_size": "10000", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_mid": "0.700000", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_quiesce_delay": "0", 2024-09-16T04:07:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_trim_interval": "1", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T04:07:58.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_auth_pins": "false", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_frag": "false", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_scatterstat": "false", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_subtrees": "false", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_default_dir_hash": "2", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_defer_session_stale": "true", 2024-09-16T04:07:58.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_max_entries": "0", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_prefetch": "true", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_early_reply": "true", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_enable_op_tracker": "true", 2024-09-16T04:07:58.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_enforce_unique_name": "true", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T04:07:58.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_health_dummy": "false", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_join_fs": "", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_journal_format": "1", 2024-09-16T04:07:58.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_create_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_export_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_import_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_journal_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_link_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_openc_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_rename_at": "0", 2024-09-16T04:07:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_shutdown_at": "0", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_event_large_threshold": "524288", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_max_events": "-1", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_max_segments": "128", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_pause": "false", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_segment_size": "0", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_trim_threshold": "128", 2024-09-16T04:07:58.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_completed_requests": "100000", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_export_size": "20971520", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_file_recover": "32", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_purge_files": "64", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_purge_ops": "8192", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T04:07:58.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_metrics_update_interval": "2", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_min_caps_per_client": "100", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_numa_node": "-1", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_duration": "600", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_size": "20", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T04:07:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_log_threshold": "5", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_ping_grace": "15", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_ping_interval": "5", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_max_caps": "30000", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T04:07:58.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_replay_interval": "1.000000", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_root_ino_gid": "0", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_root_ino_uid": "0", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T04:07:58.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_shutdown_check": "0", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_skip_ino": "0", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_snap_min_uid": "0", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_snap_rstat": "false", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_symlink_recovery": "true", 2024-09-16T04:07:58.745 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_thrash_exports": "0", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_thrash_fragments": "0", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_tick_interval": "5.000000", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_valgrind_exit": "false", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_verify_backtrace": "1", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_verify_scatter": "false", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_wipe_sessions": "false", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool_debug": "false", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T04:07:58.746 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_page_set": "false", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_page_size": "65536", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_client_bytes": "134217728", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_client_messages": "512", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_disabled_modules": "", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_enable_op_tracker": "true", 2024-09-16T04:07:58.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mds_messages": "128", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mon_messages": "128", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_op_history_duration": "600", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_op_history_size": "20", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-16T04:07:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_op_log_threshold": "5", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_osd_messages": "8192", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_pool": "true", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_standby_modules": "true", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_stats_period": "5", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_stats_threshold": "5", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_tick_period": "2", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T04:07:58.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_allow_pool_delete": "true", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_allow_pool_size_one": "true", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_bytes": "104857600", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_directed_command_retry": "5", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T04:07:58.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_target_rank": "-1", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_file_level": "debug", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T04:07:58.751 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_journald": "false", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_compact_on_start": "false", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_compact_on_trim": "true", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T04:07:58.752 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cpu_threads": "4", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data_avail_crit": "5", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data_avail_warn": "5", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_dump_json": "false", 2024-09-16T04:07:58.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_extra_checks": "false", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_require_reef": "false", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_require_squid": "false", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_down_mkfs_grace": "120", 2024-09-16T04:07:58.754 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_down_uptime_grace": "60", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_election_default_strategy": "3", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_election_timeout": "5.000000", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_enable_op_tracker": "true", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_fake_pool_delete": "false", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_force_quorum_join": "false", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T04:07:58.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_detail_to_clog": "false", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_log_update_period": "5", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_max_detail": "50", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_to_clog": "true", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_host": "172.21.15.16,172.21.15.174,172.21.15.194", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_host_override": "", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_initial_members": "", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T04:07:58.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T04:07:58.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T04:07:58.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T04:07:58.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_lease": "5.000000", 2024-09-16T04:07:58.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T04:07:58.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T04:07:58.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_log_full_interval": "50", 2024-09-16T04:07:58.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_log_max": "10000", 2024-09-16T04:07:58.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_log_max_summary": "50", 2024-09-16T04:07:58.757 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T04:07:58.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_log_epochs": "500", 2024-09-16T04:07:58.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T04:07:58.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T04:07:58.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_osd": "10000", 2024-09-16T04:07:58.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T04:07:58.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T04:07:58.761 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_memory_autotune": "true", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_memory_target": "2147483648", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_digest_period": "5", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T04:07:58.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_min_osdmap_epochs": "500", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_complaint_time": "30", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_duration": "600", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_size": "20", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_log_threshold": "5", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T04:07:58.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_cache_size": "500", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T04:07:58.764 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T04:07:58.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_report_timeout": "900", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T04:07:58.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T04:07:58.767 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_max_osds": "4", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_interval": "300", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_max_keys": "100", 2024-09-16T04:07:58.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_timeout": "300", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_session_timeout": "300", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_smart_report_timeout": "5", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_pool_size": "4", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_debug": "false", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T04:07:58.769 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_tick_interval": "5", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_use_min_delay_socket": "false", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T04:07:58.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_older_version": "true", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T04:07:58.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "monmap": "", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-16T04:07:58.772 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "motr_tracing_enabled": "false", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_op_threads": "3", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_cm": "false", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_device_name": "", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T04:07:58.773 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_sl": "3", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_type": "ib", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_reap_threshold": "5", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_before_connect": "false", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_ipv4": "true", 2024-09-16T04:07:58.774 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_ipv6": "false", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_msgr1": "true", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_msgr2": "true", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_port_max": "7568", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_port_min": "6800", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_retry_count": "3", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_retry_delay": "5", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_client": "false", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_mds": "false", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_mgr": "false", 2024-09-16T04:07:58.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_mon": "false", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_osd": "false", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_client_mode": "crc secure", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_cluster_type": "", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_compress_secure": "false", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_crc_data": "true", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_crc_header": "true", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_bug": "true", 2024-09-16T04:07:58.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_old_message": "true", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_hugepages": "", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T04:07:58.777 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_lro": "true", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_pmd": "", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_port_id": "0", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dump_on_send": "false", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-16T04:07:58.778 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_delay_type": "osd", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_network_congestion": "0", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_socket_failures": "2500", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_max_accept_failures": "4", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_max_backoff": "15.000000", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_osd_compress_mode": "none", 2024-09-16T04:07:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_pq_min_cost": "65536", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_public_type": "", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_service_mode": "crc secure", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_nodelay": "true", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_type": "async+posix", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "no_config_file": "false", 2024-09-16T04:07:58.780 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_inflight_ops": "1024", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_timeout": "10.000000", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "objectstore_blackhole": "false", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "openssl_engine_opts": "", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T04:07:58.781 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_max_ops": "4", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_slop": "0.020000", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_asio_thread_count": "2", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T04:07:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backfill_scan_max": "512", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backfill_scan_min": "64", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backoff_on_peering": "false", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_beacon_report_interval": "300", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_duration": "30", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T04:07:58.783 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_blkin_trace_all": "false", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_update_on_start": "true", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_message_cap": "256", 2024-09-16T04:07:58.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_op_priority": "63", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_watch_timeout": "30", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_command_max_records": "256", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_compact_on_start": "false", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_update_on_start": "true", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T04:07:58.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T04:07:58.786 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T04:07:58.787 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_op_order": "false", 2024-09-16T04:07:58.787 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T04:07:58.787 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T04:07:58.787 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T04:07:58.787 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T04:07:58.787 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_shutdown": "false", 2024-09-16T04:07:58.869 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T04:07:58.869 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T04:07:58.869 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_trim_objects": "false", 2024-09-16T04:07:58.869 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T04:07:58.869 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T04:07:58.869 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T04:07:58.869 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T04:07:58.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T04:07:58.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T04:07:58.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T04:07:58.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T04:07:58.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T04:07:58.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T04:07:58.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T04:07:58.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-16T04:07:58.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_default_notify_timeout": "30", 2024-09-16T04:07:58.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T04:07:58.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T04:07:58.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T04:07:58.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T04:07:58.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T04:07:58.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_enable_op_tracker": "true", 2024-09-16T04:07:58.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T04:07:58.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-16T04:07:58.872 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T04:07:58.872 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T04:07:58.872 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_info": "true", 2024-09-16T04:07:58.872 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_shutdown": "true", 2024-09-16T04:07:58.872 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T04:07:58.872 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T04:07:58.872 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T04:07:58.872 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T04:07:58.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T04:07:58.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_function_tracing": "false", 2024-09-16T04:07:58.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_grace": "20", 2024-09-16T04:07:58.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_interval": "6", 2024-09-16T04:07:58.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T04:07:58.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T04:07:58.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T04:07:58.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_stale": "600", 2024-09-16T04:07:58.874 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T04:07:58.874 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T04:07:58.874 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T04:07:58.874 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T04:07:58.874 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T04:07:58.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T04:07:58.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T04:07:58.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T04:07:58.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T04:07:58.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_journal_size": "5120", 2024-09-16T04:07:58.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_kill_backfill_at": "0", 2024-09-16T04:07:58.876 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T04:07:58.876 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_cache_size": "50", 2024-09-16T04:07:58.876 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_dedup": "true", 2024-09-16T04:07:58.876 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_message_max": "40", 2024-09-16T04:07:58.876 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T04:07:58.877 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T04:07:58.877 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_attr_name_len": "100", 2024-09-16T04:07:58.877 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_attr_size": "0", 2024-09-16T04:07:58.877 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_backfills": "1", 2024-09-16T04:07:58.877 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_markdown_count": "5", 2024-09-16T04:07:58.877 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_markdown_period": "600", 2024-09-16T04:07:58.877 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_object_name_len": "2048", 2024-09-16T04:07:58.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T04:07:58.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_object_size": "134217728", 2024-09-16T04:07:58.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T04:07:58.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T04:07:58.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T04:07:58.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T04:07:58.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T04:07:58.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pgls": "1024", 2024-09-16T04:07:58.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_push_cost": "8388608", 2024-09-16T04:07:58.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_push_objects": "10", 2024-09-16T04:07:58.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_scrubs": "3", 2024-09-16T04:07:58.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T04:07:58.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T04:07:58.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T04:07:58.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_write_size": "90", 2024-09-16T04:07:58.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T04:07:58.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T04:07:58.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T04:07:58.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T04:07:58.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T04:07:58.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T04:07:58.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T04:07:58.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T04:07:58.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_profile": "balanced", 2024-09-16T04:07:58.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T04:07:58.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_base": "805306368", 2024-09-16T04:07:58.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T04:07:58.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T04:07:58.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T04:07:58.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_target": "4294967296", 2024-09-16T04:07:58.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_target_autotune": "false", 2024-09-16T04:07:58.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T04:07:58.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T04:07:58.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_min_recovery_priority": "0", 2024-09-16T04:07:58.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T04:07:58.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_report_interval": "5", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_num_cache_shards": "32", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_numa_node": "-1", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-16T04:07:58.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objecter_finishers": "1", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objectstore": "bluestore", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objectstore_fuse": "false", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objectstore_tracing": "false", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_duration": "600", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_size": "20", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_log_threshold": "5", 2024-09-16T04:07:58.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_shards": "0", 2024-09-16T04:07:58.885 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T04:07:58.885 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T04:07:58.885 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T04:07:58.885 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T04:07:58.885 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T04:07:58.885 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T04:07:58.885 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T04:07:58.885 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T04:07:58.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T04:07:58.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T04:07:58.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_thread_timeout": "15", 2024-09-16T04:07:58.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_open_classes_on_start": "true", 2024-09-16T04:07:58.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_os_flags": "0", 2024-09-16T04:07:58.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_peering_op_priority": "255", 2024-09-16T04:07:58.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T04:07:58.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_delete_priority": "5", 2024-09-16T04:07:58.887 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T04:07:58.887 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T04:07:58.887 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T04:07:58.887 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T04:07:58.887 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T04:07:58.887 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T04:07:58.887 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T04:07:58.888 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-16T04:07:58.888 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-16T04:07:58.888 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T04:07:58.888 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T04:07:58.888 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T04:07:58.888 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T04:07:58.889 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T04:07:58.889 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T04:07:58.889 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_crimson": "false", 2024-09-16T04:07:58.889 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T04:07:58.889 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T04:07:58.889 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T04:07:58.890 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T04:07:58.890 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T04:07:58.890 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T04:07:58.890 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T04:07:58.890 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T04:07:58.890 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flags": "0", 2024-09-16T04:07:58.890 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T04:07:58.891 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_min_size": "0", 2024-09-16T04:07:58.891 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T04:07:58.891 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T04:07:58.891 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T04:07:58.891 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T04:07:58.891 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_read_ratio": "70", 2024-09-16T04:07:58.891 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_size": "2", 2024-09-16T04:07:58.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_type": "replicated", 2024-09-16T04:07:58.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T04:07:58.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T04:07:58.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T04:07:58.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T04:07:58.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T04:07:58.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T04:07:58.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_cost": "20971520", 2024-09-16T04:07:58.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T04:07:58.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_active": "0", 2024-09-16T04:07:58.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T04:07:58.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T04:07:58.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T04:07:58.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T04:07:58.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T04:07:58.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_op_priority": "3", 2024-09-16T04:07:58.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T04:07:58.894 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_priority": "5", 2024-09-16T04:07:58.894 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T04:07:58.894 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T04:07:58.894 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T04:07:58.894 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T04:07:58.894 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T04:07:58.894 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_repair_during_recovery": "false", 2024-09-16T04:07:58.895 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T04:07:58.895 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T04:07:58.895 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T04:07:58.895 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T04:07:58.895 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T04:07:58.895 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T04:07:58.896 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T04:07:58.896 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T04:07:58.896 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_chunk_max": "25", 2024-09-16T04:07:58.896 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T04:07:58.896 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_cost": "52428800", 2024-09-16T04:07:58.896 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-16T04:07:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T04:07:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_end_hour": "0", 2024-09-16T04:07:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T04:07:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T04:07:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T04:07:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T04:07:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T04:07:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T04:07:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T04:07:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T04:07:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T04:07:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_priority": "5", 2024-09-16T04:07:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T04:07:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-16T04:07:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-16T04:07:58.898 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T04:07:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_skip_data_digest": "false", 2024-09-16T04:07:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_smart_report_timeout": "5", 2024-09-16T04:07:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T04:07:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_priority": "5", 2024-09-16T04:07:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T04:07:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T04:07:58.899 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T04:07:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T04:07:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-16T04:07:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-16T04:07:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T04:07:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_target_transaction_size": "30", 2024-09-16T04:07:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T04:07:58.900 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T04:07:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T04:07:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T04:07:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T04:07:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T04:07:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T04:07:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T04:07:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T04:07:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T04:07:58.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tracing": "false", 2024-09-16T04:07:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_use_stale_snap": "false", 2024-09-16T04:07:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T04:07:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T04:07:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_kill_at": "0", 2024-09-16T04:07:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_max_join_drift": "10", 2024-09-16T04:07:58.902 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_min": "500", 2024-09-16T04:07:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_min_wait": "0.050000", 2024-09-16T04:07:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T04:07:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_service_trim_max": "500", 2024-09-16T04:07:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T04:07:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_service_trim_min": "250", 2024-09-16T04:07:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_stash_full_interval": "25", 2024-09-16T04:07:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_trim_max": "500", 2024-09-16T04:07:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_trim_min": "250", 2024-09-16T04:07:58.903 INFO:teuthology.orchestra.run.smithi016.stdout: "perf": "true", 2024-09-16T04:07:58.904 INFO:teuthology.orchestra.run.smithi016.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-16T04:07:58.904 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T04:07:58.904 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-16T04:07:58.904 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "-", 2024-09-16T04:07:58.904 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrv": "", 2024-09-16T04:07:58.904 INFO:teuthology.orchestra.run.smithi016.stdout: "public_bind_addr": "-", 2024-09-16T04:07:58.904 INFO:teuthology.orchestra.run.smithi016.stdout: "public_network": "", 2024-09-16T04:07:58.904 INFO:teuthology.orchestra.run.smithi016.stdout: "public_network_interface": "", 2024-09-16T04:07:58.905 INFO:teuthology.orchestra.run.smithi016.stdout: "qat_compressor_busy_polling": "false", 2024-09-16T04:07:58.905 INFO:teuthology.orchestra.run.smithi016.stdout: "qat_compressor_enabled": "false", 2024-09-16T04:07:58.905 INFO:teuthology.orchestra.run.smithi016.stdout: "qat_compressor_session_max_number": "256", 2024-09-16T04:07:58.905 INFO:teuthology.orchestra.run.smithi016.stdout: "rados_mon_op_timeout": "0", 2024-09-16T04:07:58.905 INFO:teuthology.orchestra.run.smithi016.stdout: "rados_osd_op_timeout": "0", 2024-09-16T04:07:58.905 INFO:teuthology.orchestra.run.smithi016.stdout: "rados_tracing": "false", 2024-09-16T04:07:58.905 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_atime_update_interval": "60", 2024-09-16T04:07:58.906 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T04:07:58.906 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T04:07:58.906 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T04:07:58.906 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T04:07:58.906 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T04:07:58.906 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T04:07:58.906 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache": "true", 2024-09-16T04:07:58.906 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T04:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T04:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T04:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T04:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_policy": "writearound", 2024-09-16T04:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_size": "33554432", 2024-09-16T04:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T04:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T04:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_compression_hint": "none", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_clone_format": "auto", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_data_pool": "", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_features": "61", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_format": "2", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_map_options": "", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_order": "22", 2024-09-16T04:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_pool": "rbd", 2024-09-16T04:07:58.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T04:07:58.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_stripe_count": "0", 2024-09-16T04:07:58.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T04:07:58.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T04:07:58.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T04:07:58.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T04:07:58.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T04:07:58.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T04:07:58.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_io_scheduler": "simple", 2024-09-16T04:07:58.910 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T04:07:58.910 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T04:07:58.910 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T04:07:58.910 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T04:07:58.910 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T04:07:58.910 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T04:07:58.910 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T04:07:58.910 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T04:07:58.911 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T04:07:58.911 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_order": "24", 2024-09-16T04:07:58.911 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_pool": "", 2024-09-16T04:07:58.911 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_splay_width": "4", 2024-09-16T04:07:58.911 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T04:07:58.911 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T04:07:58.911 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T04:07:58.911 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T04:07:58.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T04:07:58.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T04:07:58.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T04:07:58.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T04:07:58.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T04:07:58.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T04:07:58.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T04:07:58.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T04:07:58.913 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T04:07:58.913 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T04:07:58.913 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T04:07:58.913 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T04:07:58.913 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T04:07:58.913 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T04:07:58.913 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T04:07:58.913 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T04:07:58.914 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T04:07:58.914 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T04:07:58.914 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T04:07:58.914 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T04:07:58.914 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T04:07:58.914 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T04:07:58.914 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T04:07:58.914 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T04:07:58.914 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_op_threads": "1", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T04:07:58.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T04:07:58.916 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T04:07:58.916 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T04:07:58.916 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_plugins": "", 2024-09-16T04:07:58.916 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T04:07:58.916 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T04:07:58.916 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T04:07:58.916 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T04:07:58.916 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T04:07:58.917 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T04:07:58.917 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T04:07:58.917 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T04:07:58.917 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T04:07:58.917 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T04:07:58.917 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T04:07:58.917 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T04:07:58.917 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T04:07:58.917 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T04:07:58.918 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T04:07:58.918 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T04:07:58.918 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T04:07:58.918 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T04:07:58.918 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T04:07:58.918 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T04:07:58.918 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T04:07:58.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T04:07:58.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T04:07:58.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T04:07:58.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T04:07:58.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T04:07:58.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T04:07:58.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T04:07:58.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_tracing": "false", 2024-09-16T04:07:58.919 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_validate_names": "true", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_validate_pool": "true", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "restapi_base_url": "", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "restapi_log_level": "", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_admin_entry": "admin", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_backend_store": "rados", 2024-09-16T04:07:58.920 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_barbican_url": "", 2024-09-16T04:07:58.921 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_beast_enable_async": "true", 2024-09-16T04:07:58.921 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_counters_cache": "false", 2024-09-16T04:07:58.921 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-16T04:07:58.921 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T04:07:58.921 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T04:07:58.921 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T04:07:58.921 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-16T04:07:58.921 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T04:07:58.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T04:07:58.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-16T04:07:58.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cache_enabled": "true", 2024-09-16T04:07:58.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T04:07:58.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cache_lru_size": "25000", 2024-09-16T04:07:58.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_config_store": "rados", 2024-09-16T04:07:58.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_content_length_compat": "false", 2024-09-16T04:07:58.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T04:07:58.922 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T04:07:58.923 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T04:07:58.923 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cross_domain_policy": "", 2024-09-16T04:07:58.923 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T04:07:58.923 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T04:07:58.923 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T04:07:58.923 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T04:07:58.923 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T04:07:58.923 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T04:07:58.924 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T04:07:58.924 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T04:07:58.924 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T04:07:58.924 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T04:07:58.924 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T04:07:58.924 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T04:07:58.924 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T04:07:58.924 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T04:07:58.924 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T04:07:58.925 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T04:07:58.925 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T04:07:58.925 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T04:07:58.925 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T04:07:58.925 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T04:07:58.925 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T04:07:58.925 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T04:07:58.925 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T04:07:58.925 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T04:07:58.926 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T04:07:58.927 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_d4n_port": "6379", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_window": "30", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-16T04:07:58.928 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T04:07:58.929 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T04:07:58.930 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dns_name": "", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dns_s3website_name": "", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_ops_log": "false", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_static_website": "false", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_usage_log": "false", 2024-09-16T04:07:58.931 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_expose_bucket": "false", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_extended_http_attrs": "", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_filter": "none", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_objs": "32", 2024-09-16T04:07:58.932 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_init_timeout": "300", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T04:07:58.933 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_password": "", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T04:07:58.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_project": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_token": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_user": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_api_version": "2", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T04:07:58.935 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_url": "", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_objs": "32", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T04:07:58.936 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_worker": "3", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_uri": "", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T04:07:58.937 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_http_headers": "", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_attr_size": "0", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T04:07:58.938 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_listing_results": "1000", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_notify_retries": "10", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T04:07:58.939 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T04:07:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T04:07:58.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_num_control_oids": "8", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_numa_node": "-1", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_op_tracing": "false", 2024-09-16T04:07:58.942 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_opa_token": "", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_opa_url": "", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_rados": "false", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T04:07:58.943 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_posix_cache_lanes": "3", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_posix_cache_partitions": "3", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_print_continue": "true", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T04:07:58.944 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rados_tracing": "false", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_realm": "", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_realm_id": "", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_region": "", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T04:07:58.945 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_request_uri": "", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_resolve_cname": "false", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_run_sync_thread": "true", 2024-09-16T04:07:58.946 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_script_uri": "", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_service_provider_name": "", 2024-09-16T04:07:58.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_client_id": "", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_client_secret": "", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_entry": "sts", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_key": "sts", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_auth_url": "", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_custom_header": "", 2024-09-16T04:07:58.948 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_need_stats": "true", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_tenant_name": "", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_url": "", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_lease_period": "120", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T04:07:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_thread_pool_size": "512", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-16T04:07:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_comment": "", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_createby": "", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_encoding": "", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_flag": "false", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_origin": "", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_tracker": "", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T04:07:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_max_shards": "32", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_use_opa_authz": "false", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_counters_cache": "false", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_policies_max_num": "100", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T04:07:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_unique_email": "true", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_verify_ssl": "true", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zone": "", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zone_id": "", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zonegroup": "", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zonegroup_id": "", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T04:07:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "rgwlc_auto_session_clear": "true", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_block_size": "4096", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T04:07:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_partition_filters": "false", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_perf": "false", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T04:07:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "run_dir": "/var/run/ceph", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_block_create": "true", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_device_size": "53687091200", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_full_integrity_check": "false", 2024-09-16T04:07:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_main_device_type": "SSD", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "seastore_segment_size": "67108864", 2024-09-16T04:07:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "set_keepcaps": "false", 2024-09-16T04:07:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "setgroup": "", 2024-09-16T04:07:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "setuser": "", 2024-09-16T04:07:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "setuser_match_path": "", 2024-09-16T04:07:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T04:07:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "thp": "false", 2024-09-16T04:07:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "threadpool_default_timeout": "60", 2024-09-16T04:07:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T04:07:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "throttler_perf_counter": "true" 2024-09-16T04:07:58.958 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:07:58.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T04:07:59.096 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a: {} 2024-09-16T04:07:59.096 INFO:teuthology.orchestra.run.smithi016.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:07:59.110 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b: {} 2024-09-16T04:07:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:07:59.125 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c: {} 2024-09-16T04:07:59.125 INFO:teuthology.orchestra.run.smithi016.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:07:59.136 DEBUG:teuthology.orchestra.run.smithi016:> type ceph-objectstore-tool 2024-09-16T04:07:59.151 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:07:59.151 DEBUG:teuthology.orchestra.run.smithi174:> type ceph-objectstore-tool 2024-09-16T04:07:59.170 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:07:59.171 DEBUG:teuthology.orchestra.run.smithi194:> type ceph-objectstore-tool 2024-09-16T04:07:59.189 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:07:59.190 INFO:teuthology.run_tasks:Running task rados... 2024-09-16T04:07:59.201 INFO:tasks.rados:Beginning rados... 2024-09-16T04:07:59.201 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-16T04:07:59.209 INFO:tasks.rados:joining rados 2024-09-16T04:07:59.209 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T04:07:59.209 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T04:07:59.209 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:07:59.210 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T04:07:59.210 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T04:07:59.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:07:59.211 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T04:07:59.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:07:59.213 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T04:07:59.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:07:59.213 INFO:tasks.rados:clients are ['client.0'] 2024-09-16T04:07:59.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lrcprofile crush-failure-domain=osd k=4 l=3 m=2 name=lrcprofile plugin=lrc 2024-09-16T04:07:59.215 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:07:59.210+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:07:59.312 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:07:59.310+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:07:59.331 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:07:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-16T04:07:59.331 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-16T04:07:59.331 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:07:59.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:07:59.367 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:07:59.367 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:07:59.406 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:07:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:07:59.413 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:07:59.411+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:07:59.443 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:07:59.444 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:07:59.452 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:07:59.453 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:07:59.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:07:59.480 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:07:59.481 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:07:59.514 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:07:59.512+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:07:59.520 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:07:59.520 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:07:59.559 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:07:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:07:59.588 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:07:59.589 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:07:59.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:07:59.598 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:07:59.599 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:07:59.639 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:07:59.639 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:07:59.639 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:07:59.637+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:07:59.679 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:07:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:07:59.703 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:07:59.703 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:07:59.706 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:07:59.706 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:07:59.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:07:59.716 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:07:59.714+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:07:59.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:07:59.725 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:07:59.747 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:07:59.748 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:07:59.816 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:07:59.815+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:07:59.831 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:07:59.831 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:07:59.840 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:07:59.917 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:07:59.915+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:07:59.947 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:07:59.947 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:07:59.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:00.040 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:00.038+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:00.048 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-16T04:08:00.064 INFO:tasks.rados:starting run 0 out of 1 2024-09-16T04:08:00.064 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-16T04:08:00.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lrcprofile 2024-09-16T04:08:00.106 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:00.107 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:00.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:00.156 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:00.154+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:00.220 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:00.218+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:00.266 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:00.266 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:00.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:00.320 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:00.319+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:00.384 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:00.384 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:00.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:00.422 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:00.420+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:00.523 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:00.521+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:00.540 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:00.540 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:00.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:00.625 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:00.623+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:00.656 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:00.656 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:00.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:00.725 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:00.723+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:00.773 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:00.773 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:00.783 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:00.826 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:00.824+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:00.898 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:00.898 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:00.907 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:00.927 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:00.925+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:01.010 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:01.010 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:01.023 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:01.067 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:01.066+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:01.127 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:01.127 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:01.128 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:01.127+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:01.135 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:01.230 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:01.228+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:01.278 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:01.278 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:01.287 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:01.331 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:01.329+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:01.389 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:01.389 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:01.398 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:01.440 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:01.438+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:01.532 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:01.531+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:01.542 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:01.542 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:01.550 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:01.637 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:01.632+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:01.654 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:01.654 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:01.663 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:01.735 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:01.733+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:01.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:01.767 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:01.767 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:01.775 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:01.836 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:01.834+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:01.905 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:01.920 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:01.920 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:01.927 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:01.927 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:01.930 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:01.936 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:01.935+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:01.948 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:01.969 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:01.990 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:02.012 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:02.012 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:02.032 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:02.034 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:02.035 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:02.038 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:02.036+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:02.047 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'unique_pool_0' created 2024-09-16T04:08:02.054 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:02.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:08:02.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-16T04:08:02.079 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:02.079 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:02.102 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:02.139 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:02.137+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:02.140 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:02.141 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:02.154 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:02.154 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:02.154 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:02.154 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:02.155 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:02.163 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:02.168 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:02.168 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:02.239 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:02.238+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:02.308 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:02.308 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:02.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:02.308 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:02.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.364047342999999, 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.309 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:02.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:02.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:02.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923042+0000", 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 11.381728638, 2024-09-16T04:08:02.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075287749999999997, 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923042+0000", 2024-09-16T04:08:02.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923042+0000", 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923042+0000", 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2700000000000001e-07 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923097+0000", 2024-09-16T04:08:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4897e-05 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923103+0000", 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0299999999999999e-06 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923106+0000", 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.591e-06 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.928324+0000", 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005217742 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.928428+0000", 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010402499999999999 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.928560+0000", 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000131698 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930427+0000", 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001867226 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930470+0000", 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3241999999999997e-05 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930529+0000", 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8576999999999998e-05 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930570+0000", 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1520000000000002e-05 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923113+0000", 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 11.381657684, 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0076822339999999996, 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.321 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923113+0000", 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923113+0000", 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923113+0000", 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4999999999999998e-07 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923115+0000", 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4420000000000001e-06 2024-09-16T04:08:02.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923120+0000", 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.211e-06 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923123+0000", 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.332e-06 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.929534+0000", 2024-09-16T04:08:02.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0064114539999999996 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.929635+0000", 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000100299 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.929770+0000", 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000135351 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930683+0000", 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00091337999999999999 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930724+0000", 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1063999999999999e-05 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930757+0000", 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2376999999999998e-05 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930795+0000", 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7973999999999999e-05 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 11.3816431, 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.327 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.328 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.328 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.328 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.328 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:02.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.328 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:02.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:02.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:02.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:02.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:02.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:02.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:02.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:02.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:02.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:02.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.335 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 11.371280594, 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:02.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:57 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:51.800186+0000", 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.504584062999999, 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075740590000000002, 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.340 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 57 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800186+0000", 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.341 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800186+0000", 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800187+0000", 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1650000000000001e-06 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800225+0000", 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7923999999999998e-05 2024-09-16T04:08:02.342 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800236+0000", 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0658e-05 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800243+0000", 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0389999999999999e-06 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800281+0000", 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7764e-05 2024-09-16T04:08:02.343 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800323+0000", 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2256000000000003e-05 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800395+0000", 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1962e-05 2024-09-16T04:08:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.802001+0000", 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016061719999999999 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.807705+0000", 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057043279999999998 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.345 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.807737+0000", 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1322999999999998e-05 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.807760+0000", 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3468e-05 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:58 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:02.346 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:51.800265+0000", 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.504505483999999, 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086104170000000004, 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 58 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.347 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800265+0000", 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800265+0000", 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800265+0000", 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0100000000000004e-07 2024-09-16T04:08:02.348 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.349 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.349 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.349 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800353+0000", 2024-09-16T04:08:02.349 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7968999999999998e-05 2024-09-16T04:08:02.349 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.349 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800363+0000", 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.815e-06 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800371+0000", 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5220000000000001e-06 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800725+0000", 2024-09-16T04:08:02.350 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035470800000000002 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800776+0000", 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0625999999999999e-05 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800859+0000", 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2812999999999995e-05 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.351 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.802211+0000", 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013517519999999999 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.808815+0000", 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0066041909999999997 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.352 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.808850+0000", 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4761999999999997e-05 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.808875+0000", 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5558000000000002e-05 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.353 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:51.800386+0000", 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.504383816000001, 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086667150000000002, 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 59 2024-09-16T04:08:02.354 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.355 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.355 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.355 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800386+0000", 2024-09-16T04:08:02.355 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.355 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.355 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.355 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800386+0000", 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800387+0000", 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9999999999999997e-07 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800392+0000", 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.899e-06 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800400+0000", 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5410000000000003e-06 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800405+0000", 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.823e-06 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.801125+0000", 2024-09-16T04:08:02.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00071925799999999996 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.801175+0000", 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0037999999999998e-05 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.801263+0000", 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8392000000000003e-05 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.802218+0000", 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095508799999999999 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.809001+0000", 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067828769999999997 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.809031+0000", 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9920000000000002e-05 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.809053+0000", 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2178999999999999e-05 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:02.360 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:51.825046+0000", 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.479724158, 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082050490000000007, 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 88 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.361 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825046+0000", 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825046+0000", 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.362 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825046+0000", 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5699999999999998e-07 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825049+0000", 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0130000000000002e-06 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825054+0000", 2024-09-16T04:08:02.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6090000000000003e-06 2024-09-16T04:08:02.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.364 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825057+0000", 2024-09-16T04:08:02.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2109999999999998e-06 2024-09-16T04:08:02.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.365 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.827954+0000", 2024-09-16T04:08:02.365 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002897125 2024-09-16T04:08:02.365 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.366 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.828005+0000", 2024-09-16T04:08:02.366 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0847999999999998e-05 2024-09-16T04:08:02.366 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.366 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.366 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.367 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.828085+0000", 2024-09-16T04:08:02.367 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9419000000000007e-05 2024-09-16T04:08:02.367 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.367 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.367 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.367 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.828558+0000", 2024-09-16T04:08:02.368 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047363100000000002 2024-09-16T04:08:02.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.368 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.368 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.833195+0000", 2024-09-16T04:08:02.368 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004636444 2024-09-16T04:08:02.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.833227+0000", 2024-09-16T04:08:02.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.201e-05 2024-09-16T04:08:02.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.370 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.370 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.370 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.833251+0000", 2024-09-16T04:08:02.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4382000000000001e-05 2024-09-16T04:08:02.371 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.371 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.371 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.371 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.371 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.371 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:02.372 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:02.372 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.874692542, 2024-09-16T04:08:02.372 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:02.372 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.372 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.372 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.373 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.373 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.373 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.373 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:02.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.373 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.373 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.374 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:02.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:02.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.375 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:02.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:02.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.376 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:02.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:02.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:02.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:02.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:02.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:02.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:02.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:02.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.380 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:02.380 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:02.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.380 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.380 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:02.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:02.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:02.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:02.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.382 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:02.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:02.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:02.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:02.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.384 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:02.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:02.384 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.384 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.384 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.385 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.385 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.453512+0000", 2024-09-16T04:08:02.385 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8512578249999994, 2024-09-16T04:08:02.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069585710000000002, 2024-09-16T04:08:02.385 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.386 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.386 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.386 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.386 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.386 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.386 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-09-16T04:08:02.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.387 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453512+0000", 2024-09-16T04:08:02.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453512+0000", 2024-09-16T04:08:02.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.389 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.389 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453513+0000", 2024-09-16T04:08:02.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:02.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453565+0000", 2024-09-16T04:08:02.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1987000000000002e-05 2024-09-16T04:08:02.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453573+0000", 2024-09-16T04:08:02.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4309999999999994e-06 2024-09-16T04:08:02.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453580+0000", 2024-09-16T04:08:02.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5069999999999997e-06 2024-09-16T04:08:02.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.458788+0000", 2024-09-16T04:08:02.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005207761 2024-09-16T04:08:02.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.458832+0000", 2024-09-16T04:08:02.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3890000000000002e-05 2024-09-16T04:08:02.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.458904+0000", 2024-09-16T04:08:02.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2004999999999999e-05 2024-09-16T04:08:02.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.394 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.394 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459640+0000", 2024-09-16T04:08:02.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00073623500000000001 2024-09-16T04:08:02.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460428+0000", 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00078837600000000003 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460453+0000", 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4851999999999999e-05 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460471+0000", 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7972e-05 2024-09-16T04:08:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8511796399999998, 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092583400000000003, 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.396 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.37e-07 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.398 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453604+0000", 2024-09-16T04:08:02.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2658e-05 2024-09-16T04:08:02.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453611+0000", 2024-09-16T04:08:02.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0650000000000001e-06 2024-09-16T04:08:02.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453616+0000", 2024-09-16T04:08:02.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9099999999999996e-06 2024-09-16T04:08:02.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459323+0000", 2024-09-16T04:08:02.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057072879999999996 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459369+0000", 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6436000000000001e-05 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459442+0000", 2024-09-16T04:08:02.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2459000000000005e-05 2024-09-16T04:08:02.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.402 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460087+0000", 2024-09-16T04:08:02.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064517999999999997 2024-09-16T04:08:02.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.402 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462796+0000", 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027087639999999998 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462827+0000", 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0827000000000001e-05 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462849+0000", 2024-09-16T04:08:02.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2215999999999999e-05 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4203834509999993, 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088271749999999996, 2024-09-16T04:08:02.404 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 158 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5899999999999996e-07 2024-09-16T04:08:02.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.407 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884453+0000", 2024-09-16T04:08:02.407 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999997e-05 2024-09-16T04:08:02.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.408 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884462+0000", 2024-09-16T04:08:02.408 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2920000000000001e-06 2024-09-16T04:08:02.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.408 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.408 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.408 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884468+0000", 2024-09-16T04:08:02.408 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9399999999999999e-06 2024-09-16T04:08:02.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886225+0000", 2024-09-16T04:08:02.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0017572079999999999 2024-09-16T04:08:02.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886269+0000", 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3118999999999998e-05 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886334+0000", 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5695e-05 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888544+0000", 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002210086 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893152+0000", 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046078940000000004 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893177+0000", 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.463e-05 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893214+0000", 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7002000000000002e-05 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.411 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.412 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.412 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.412 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:02.412 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4202884220000005, 2024-09-16T04:08:02.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087992620000000004, 2024-09-16T04:08:02.412 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.412 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.413 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.413 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.413 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.413 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.413 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-09-16T04:08:02.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.413 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.413 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:02.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:02.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:02.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0500000000000003e-07 2024-09-16T04:08:02.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884486+0000", 2024-09-16T04:08:02.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-06 2024-09-16T04:08:02.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884493+0000", 2024-09-16T04:08:02.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8759999999999999e-06 2024-09-16T04:08:02.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884498+0000", 2024-09-16T04:08:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3550000000000004e-06 2024-09-16T04:08:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888576+0000", 2024-09-16T04:08:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040780230000000001 2024-09-16T04:08:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888629+0000", 2024-09-16T04:08:02.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2892000000000001e-05 2024-09-16T04:08:02.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888704+0000", 2024-09-16T04:08:02.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5024000000000005e-05 2024-09-16T04:08:02.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.419 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892724+0000", 2024-09-16T04:08:02.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040194619999999997 2024-09-16T04:08:02.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893222+0000", 2024-09-16T04:08:02.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049801700000000001 2024-09-16T04:08:02.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893246+0000", 2024-09-16T04:08:02.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3901e-05 2024-09-16T04:08:02.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893281+0000", 2024-09-16T04:08:02.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5277000000000001e-05 2024-09-16T04:08:02.421 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.421 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.421 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4202603870000008, 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088385849999999995, 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.422 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 160 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3900000000000005e-07 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884514+0000", 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2169999999999999e-06 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884520+0000", 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8639999999999998e-06 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884527+0000", 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.866e-06 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888915+0000", 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043874270000000002 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888953+0000", 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7839000000000001e-05 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889014+0000", 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1289000000000002e-05 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892771+0000", 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037575199999999999 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893289+0000", 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00051718599999999999 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893314+0000", 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5202999999999999e-05 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893348+0000", 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4635000000000003e-05 2024-09-16T04:08:02.427 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4202232069999994, 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088608530000000001, 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.428 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.429 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.429 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.429 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.429 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.429 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 161 2024-09-16T04:08:02.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.429 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:02.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.430 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.430 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:02.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.431 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.431 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884548+0000", 2024-09-16T04:08:02.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5599999999999995e-07 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884603+0000", 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5615999999999997e-05 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884612+0000", 2024-09-16T04:08:02.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6589999999999996e-06 2024-09-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884619+0000", 2024-09-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4630000000000004e-06 2024-09-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889203+0000", 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045841989999999997 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889242+0000", 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8713e-05 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889304+0000", 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1858000000000003e-05 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892819+0000", 2024-09-16T04:08:02.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003515107 2024-09-16T04:08:02.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893354+0000", 2024-09-16T04:08:02.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053514000000000003 2024-09-16T04:08:02.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.436 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.436 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893377+0000", 2024-09-16T04:08:02.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2265000000000001e-05 2024-09-16T04:08:02.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893408+0000", 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1276999999999999e-05 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4201439580000006, 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094233230000000008, 2024-09-16T04:08:02.437 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 162 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884627+0000", 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884629+0000", 2024-09-16T04:08:02.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.187e-06 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884632+0000", 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.501e-06 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884635+0000", 2024-09-16T04:08:02.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7389999999999999e-06 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892223+0000", 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075881029999999997 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892291+0000", 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7508000000000005e-05 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892412+0000", 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012124 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893979+0000", 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015676069999999999 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893987+0000", 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.306e-06 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894013+0000", 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6305999999999999e-05 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894050+0000", 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6492999999999998e-05 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.443 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4201315870000002, 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095013200000000006, 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 163 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2100000000000001e-07 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884642+0000", 2024-09-16T04:08:02.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.478e-06 2024-09-16T04:08:02.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884646+0000", 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.292e-06 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884648+0000", 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6230000000000001e-06 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892847+0000", 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0081990580000000004 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892891+0000", 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3628999999999999e-05 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892959+0000", 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8151000000000005e-05 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894055+0000", 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001095927 2024-09-16T04:08:02.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894088+0000", 2024-09-16T04:08:02.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3016999999999999e-05 2024-09-16T04:08:02.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894117+0000", 2024-09-16T04:08:02.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9204999999999999e-05 2024-09-16T04:08:02.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894140+0000", 2024-09-16T04:08:02.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2719000000000001e-05 2024-09-16T04:08:02.450 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.450 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.450 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4201180489999992, 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.451 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:02.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:02.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4200639880000008, 2024-09-16T04:08:02.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:02.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:02.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:02.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:02.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:02.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:02.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:02.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:02.463 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.464 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.464 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.464 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:02.464 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:02.464 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:02.465 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:02.466 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:02.463+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:02.566 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:02.564+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:02.580 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:02.580 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:02.580 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:02.580 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:02.589 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:02.666 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:02.665+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:02.693 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:02.693 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:02.693 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:02.693 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:02.693 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:02.702 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:02.767 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:02.766+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:02.805 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:02.806 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:02.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:02.806 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [] 2024-09-16T04:08:02.806 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:02.814 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:02.868 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:02.867+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:02.917 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:02.917 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:02.917 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:02.917 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:02.926 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:02.969 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:02.968+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:03.028 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:03.028 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:03.028 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:03.028 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:03.029 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:03.037 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:03.064 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-16T04:08:03.064 INFO:teuthology.orchestra.run.smithi016.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-16T04:08:03.071 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:03.069+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:03.172 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:03.170+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:03.180 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [] 2024-09-16T04:08:03.180 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:03.189 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:03.273 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:03.271+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:03.300 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:03.300 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:03.300 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:03.300 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:03.308 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:03.373 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:03.372+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:03.451 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:03.451 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:03.451 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:03.452 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:03.452 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:03.460 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:03.503 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:03.502+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:03.563 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:03.563 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:03.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:03.563 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [] 2024-09-16T04:08:03.563 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:03.571 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:03.575 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:03.573+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:03.676 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:03.675+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:03.715 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:03.715 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:03.723 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:03.777 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:03.775+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:03.826 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:03.826 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:03.826 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:03.826 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:03.826 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:03.834 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:03.878 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:03.876+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:03.939 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:03.939 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:03.939 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:03.939 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:03.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.939 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:37 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'23, mlcod=20'19)", 2024-09-16T04:08:03.939 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:50.930360+0000", 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 13.005869477999999, 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030961579999999999, 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.930360+0000", 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.930360+0000", 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.930361+0000", 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6899999999999996e-07 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.930367+0000", 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2179999999999996e-06 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.930390+0000", 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2855000000000001e-05 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.930395+0000", 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6160000000000004e-06 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.930433+0000", 2024-09-16T04:08:03.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8062999999999998e-05 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.930446+0000", 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2848e-05 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.933410+0000", 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029644990000000002 2024-09-16T04:08:03.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.944 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.944 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:50.933457+0000", 2024-09-16T04:08:03.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6289999999999999e-05 2024-09-16T04:08:03.944 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.944 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.944 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.944 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:87 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'57, mlcod=21'52)", 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.827314+0000", 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.108916201, 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029541799999999998, 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.827314+0000", 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.827314+0000", 2024-09-16T04:08:03.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.827314+0000", 2024-09-16T04:08:03.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6599999999999996e-07 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.827377+0000", 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3254000000000001e-05 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.827400+0000", 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2484999999999999e-05 2024-09-16T04:08:03.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.827406+0000", 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7740000000000001e-06 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.827437+0000", 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1035999999999999e-05 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.948 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.827452+0000", 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5557999999999999e-05 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.830238+0000", 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027857419999999999 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.830268+0000", 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9765e-05 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.949 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:88 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'58, mlcod=21'54)", 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.828181+0000", 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.108049210000001, 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048948150000000003, 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828181+0000", 2024-09-16T04:08:03.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828181+0000", 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828181+0000", 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0399999999999996e-07 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828185+0000", 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.467e-06 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828202+0000", 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7147e-05 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828207+0000", 2024-09-16T04:08:03.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1390000000000003e-06 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828237+0000", 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0386e-05 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828252+0000", 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4567e-05 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.953 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.833042+0000", 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047899819999999999 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.833075+0000", 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3623e-05 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:105 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'67, mlcod=21'64)", 2024-09-16T04:08:03.954 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.998266+0000", 2024-09-16T04:08:03.955 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.937964161, 2024-09-16T04:08:03.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030726199999999999, 2024-09-16T04:08:03.955 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.955 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.955 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.955 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998266+0000", 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998266+0000", 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998266+0000", 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.99e-07 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998271+0000", 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.566e-06 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998296+0000", 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5571e-05 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998304+0000", 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9719999999999992e-06 2024-09-16T04:08:03.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998340+0000", 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5389000000000003e-05 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998358+0000", 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8459999999999999e-05 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001304+0000", 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029461660000000001 2024-09-16T04:08:03.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001338+0000", 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3896999999999997e-05 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.959 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:106 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=21'64)", 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.998678+0000", 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.937551419, 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0052358589999999998, 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998678+0000", 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.960 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998678+0000", 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998679+0000", 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1600000000000001e-07 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998722+0000", 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2456999999999997e-05 2024-09-16T04:08:03.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.962 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.962 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998748+0000", 2024-09-16T04:08:03.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6679999999999999e-05 2024-09-16T04:08:03.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.962 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.962 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998754+0000", 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1659999999999999e-06 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998793+0000", 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8903999999999999e-05 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998808+0000", 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4724e-05 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003876+0000", 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050678140000000003 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003914+0000", 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8398000000000001e-05 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.964 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:107 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=21'64)", 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.999067+0000", 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.937163346, 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048815799999999999, 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999067+0000", 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.965 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999067+0000", 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999067+0000", 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4600000000000004e-07 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999073+0000", 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8409999999999996e-06 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999099+0000", 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5936e-05 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999105+0000", 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7509999999999997e-06 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999139+0000", 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4230000000000003e-05 2024-09-16T04:08:03.967 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999152+0000", 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2713e-05 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003917+0000", 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004765516 2024-09-16T04:08:03.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003948+0000", 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0846999999999998e-05 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:108 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=21'64)", 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.999445+0000", 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.936785214, 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045202180000000003, 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.969 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999445+0000", 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999445+0000", 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999449+0000", 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4109999999999998e-06 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999454+0000", 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7670000000000003e-06 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999474+0000", 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0111000000000001e-05 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999479+0000", 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2560000000000002e-06 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999519+0000", 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9672999999999998e-05 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999532+0000", 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3040999999999999e-05 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003949+0000", 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044173839999999999 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003965+0000", 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5574999999999999e-05 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.973 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:109 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=21'65)", 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.999872+0000", 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.936357898000001, 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041208269999999996, 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.974 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999872+0000", 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999872+0000", 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999872+0000", 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1699999999999999e-07 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999940+0000", 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7838999999999999e-05 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999960+0000", 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0134999999999999e-05 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999966+0000", 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5230000000000003e-06 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000005+0000", 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8994e-05 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000021+0000", 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6192e-05 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003967+0000", 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039464169999999998 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003993+0000", 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.531e-05 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:110 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=21'65)", 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.000242+0000", 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.935988350000001, 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037826499999999998, 2024-09-16T04:08:03.978 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000242+0000", 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000242+0000", 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.979 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000242+0000", 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2699999999999998e-07 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000248+0000", 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9440000000000002e-06 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000273+0000", 2024-09-16T04:08:03.980 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4593999999999999e-05 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000278+0000", 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4399999999999996e-06 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000315+0000", 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7073999999999998e-05 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000328+0000", 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2879e-05 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003994+0000", 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003665705 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004024+0000", 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0187000000000001e-05 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.982 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:111 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'73, mlcod=21'65)", 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.000585+0000", 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.935645235999999, 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034651180000000001, 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000585+0000", 2024-09-16T04:08:03.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000585+0000", 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000585+0000", 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1399999999999997e-07 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.984 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000589+0000", 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0249999999999996e-06 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000608+0000", 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8479e-05 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000614+0000", 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9179999999999997e-06 2024-09-16T04:08:03.985 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000666+0000", 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1103000000000002e-05 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000678+0000", 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2775999999999999e-05 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004026+0000", 2024-09-16T04:08:03.986 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0033480960000000001 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004050+0000", 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3306999999999999e-05 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:112 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'74, mlcod=21'65)", 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.000915+0000", 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.935314903, 2024-09-16T04:08:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003398014, 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000915+0000", 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000915+0000", 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.988 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000916+0000", 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6899999999999997e-07 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000989+0000", 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3436999999999995e-05 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001010+0000", 2024-09-16T04:08:03.989 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1248999999999999e-05 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001016+0000", 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8429999999999998e-06 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001066+0000", 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0247000000000002e-05 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.990 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001082+0000", 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5960999999999998e-05 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004290+0000", 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032079360000000002 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004313+0000", 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2671999999999999e-05 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.991 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:113 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'75, mlcod=21'65)", 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.001303+0000", 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.934926883999999, 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030327570000000001, 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001303+0000", 2024-09-16T04:08:03.992 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001303+0000", 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001304+0000", 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7999999999999995e-07 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.993 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001310+0000", 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2999999999999998e-06 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001336+0000", 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5709999999999999e-05 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001341+0000", 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7950000000000004e-06 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001375+0000", 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3957e-05 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001389+0000", 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.311e-05 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004315+0000", 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029261059999999999 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.995 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004336+0000", 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1098999999999999e-05 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:114 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'76, mlcod=21'65)", 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.001687+0000", 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.934542845999999, 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026669380000000002, 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:03.996 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001687+0000", 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001687+0000", 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:03.997 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001687+0000", 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.03e-07 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001695+0000", 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2049999999999996e-06 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001714+0000", 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9768000000000001e-05 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.998 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001720+0000", 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5219999999999998e-06 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001755+0000", 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5049999999999998e-05 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001768+0000", 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2761e-05 2024-09-16T04:08:03.999 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004337+0000", 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025695710000000001 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004354+0000", 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6657999999999999e-05 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.000 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:119 1.0 e21/19 1:30306672:devicehealth::main.db.0000000000000000:head v 21'81, mlcod=21'78)", 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.006196+0000", 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.930033558, 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027590570000000001, 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006196+0000", 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.001 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006196+0000", 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006197+0000", 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2500000000000005e-07 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:04.002 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006262+0000", 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5234000000000006e-05 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006290+0000", 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8305000000000001e-05 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006299+0000", 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6640000000000004e-06 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.003 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006341+0000", 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1458999999999997e-05 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006363+0000", 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2371999999999999e-05 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.008918+0000", 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025552389999999999 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.004 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.008955+0000", 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7159000000000001e-05 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:120 1.0 e21/19 1:30306672:devicehealth::main.db.0000000000000000:head v 21'82, mlcod=21'78)", 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.006663+0000", 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.929567027000001, 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002479649, 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:04.005 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006663+0000", 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006663+0000", 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:04.006 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006663+0000", 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1500000000000005e-07 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006707+0000", 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3823999999999999e-05 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006726+0000", 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9312000000000001e-05 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.007 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006732+0000", 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5140000000000001e-06 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006803+0000", 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0582999999999998e-05 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:04.008 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.006815+0000", 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2599000000000001e-05 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.009120+0000", 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00230474 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.009142+0000", 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2561999999999998e-05 2024-09-16T04:08:04.009 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:121 1.0 e21/19 1:30306672:devicehealth::main.db.0000000000000000:head v 21'83, mlcod=21'78)", 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.007079+0000", 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.929150874999999, 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0024797140000000001, 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.010 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.007079+0000", 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.007079+0000", 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.007080+0000", 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8100000000000002e-07 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.011 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.007150+0000", 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0462000000000004e-05 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.007185+0000", 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4807000000000001e-05 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.007191+0000", 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3840000000000002e-06 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.012 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.007229+0000", 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8049000000000001e-05 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.007244+0000", 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.456e-05 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:04.013 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.009540+0000", 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022963660000000002 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.009559+0000", 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8304999999999998e-05 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:124 1.0 e22/19 1:30306672:devicehealth::main.db.0000000000000000:head v 22'86, mlcod=21'84)", 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.505448643999999, 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020095396000000001, 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:04.016 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:04.016 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.016 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.016 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.016 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:04.016 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430782+0000", 2024-09-16T04:08:04.016 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9699999999999996e-07 2024-09-16T04:08:04.016 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.016 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.017 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:04.017 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430789+0000", 2024-09-16T04:08:04.017 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4660000000000002e-06 2024-09-16T04:08:04.017 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.017 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450028+0000", 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019238816999999998 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450042+0000", 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4487e-05 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450169+0000", 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126787 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450188+0000", 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450816+0000", 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062741399999999997 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450877+0000", 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0782999999999997e-05 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:136 1.0 e22/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 22'94, mlcod=22'88)", 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.457035+0000", 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.479194468999999, 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002872379, 2024-09-16T04:08:04.020 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457035+0000", 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457035+0000", 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.021 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457036+0000", 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0999999999999997e-07 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457043+0000", 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8519999999999998e-06 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457077+0000", 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3939000000000002e-05 2024-09-16T04:08:04.022 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457095+0000", 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8482e-05 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457139+0000", 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3832999999999999e-05 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:04.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457152+0000", 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2201000000000001e-05 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459879+0000", 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027271919999999998 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459908+0000", 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9070000000000001e-05 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:04.024 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:137 1.0 e22/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 22'95, mlcod=22'91)", 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.457597+0000", 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.47863328, 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025113850000000001, 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457597+0000", 2024-09-16T04:08:04.025 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457597+0000", 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457597+0000", 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:04.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457670+0000", 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3059999999999995e-05 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457701+0000", 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.082e-05 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457709+0000", 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6499999999999996e-06 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:04.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457750+0000", 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0886000000000001e-05 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.457763+0000", 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2891000000000001e-05 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.460070+0000", 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023070209999999998 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.460108+0000", 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8266999999999998e-05 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:141 1.0 e22/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 22'99, mlcod=22'92)", 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.459547+0000", 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.476682509, 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003111295, 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:04.029 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459547+0000", 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459547+0000", 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:04.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459548+0000", 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6599999999999994e-07 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459556+0000", 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.43e-06 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459595+0000", 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.913e-05 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:04.031 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459603+0000", 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7929999999999998e-06 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459658+0000", 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5424000000000003e-05 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.459673+0000", 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4992e-05 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.462615+0000", 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029422369999999999 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.462659+0000", 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3322999999999998e-05 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:04.033 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:04.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:04.035 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:04.033+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:04.093 DEBUG:teuthology.orchestra.run.smithi016:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-09-16T04:08:04.124 INFO:tasks.rados.rados.0.smithi016.stdout:adding op weight read -> 100 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:adding op weight delete -> 50 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:adding op weight snap_create -> 50 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:adding op weight snap_remove -> 50 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:adding op weight rollback -> 50 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:adding op weight setattr -> 25 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:adding op weight rmattr -> 25 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:adding op weight copy_from -> 50 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:adding op weight append -> 100 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout:Configuration: 2024-09-16T04:08:04.125 INFO:tasks.rados.rados.0.smithi016.stdout: Number of operations: 400 2024-09-16T04:08:04.126 INFO:tasks.rados.rados.0.smithi016.stdout: Number of objects: 50 2024-09-16T04:08:04.126 INFO:tasks.rados.rados.0.smithi016.stdout: Max in flight operations: 16 2024-09-16T04:08:04.126 INFO:tasks.rados.rados.0.smithi016.stdout: Object size (in bytes): 4000000 2024-09-16T04:08:04.126 INFO:tasks.rados.rados.0.smithi016.stdout: Write stride min: 400000 2024-09-16T04:08:04.126 INFO:tasks.rados.rados.0.smithi016.stdout: Write stride max: 800000 2024-09-16T04:08:04.135 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:04.133+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:04.157 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:08:04.157 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-16T04:08:04.157 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-16T04:08:04.157 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:08:04.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:04.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:04.210 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T04:08:04.210 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:08:04.210 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T04:08:04.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T04:08:04.236 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:04.234+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:04.318 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:08:04.318 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-16T04:08:04.318 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-16T04:08:04.318 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-16T04:08:04.318 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:08:04.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:04.336 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:04.335+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:04.343 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:04.343 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:04.366 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:04.366 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:04.397 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:04.398 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:04.418 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:04.437 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:04.435+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:04.438 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:04.439 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:04.441 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:04.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:08:04.462 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:04.484 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:04.484 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:04.506 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:04.507 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:04.528 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:04.539 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:04.537+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:04.550 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 29, 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "44381953-4509-4b15-853e-3887928dfa29", 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2024-09-16T04:07:36.650572+0000", 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2024-09-16T04:08:04.062296+0000", 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2024-09-16T04:07:48.919317+0000", 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2024-09-16T04:07:39.793439+0000", 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T04:08:04.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5806080, 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: "nodeep-scrub", 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: "noscrub", 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 7, 2024-09-16T04:08:04.570 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 3, 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 12, 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "jewel", 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "squid", 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_crimson": false, 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2024-09-16T04:08:04.571 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": ".mgr", 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-16T04:07:48.723300+0000", 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "is_stretch_pool": false, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-16T04:08:04.572 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-16T04:08:04.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "21", 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-16T04:08:04.574 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2024-09-16T04:08:04.576 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": {} 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "score_type": "Fair distribution", 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 11.760000228881836, 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 11.760000228881836, 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.17000000178813934, 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2, 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2, 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-16T04:08:04.577 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-16T04:07:49.309634+0000", 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-16T04:08:04.578 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "is_stretch_pool": false, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-16T04:08:04.579 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "23", 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 2, 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 23, 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-16T04:08:04.580 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-16T04:08:04.581 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-16T04:08:04.582 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "score_type": "Fair distribution", 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 3, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 3, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.75, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2.25, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2.25, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-16T04:08:04.583 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.584 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 3, 2024-09-16T04:08:04.584 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "unique_pool_0", 2024-09-16T04:08:04.584 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-16T04:08:01.082632+0000", 2024-09-16T04:08:04.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-16T04:08:04.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2024-09-16T04:08:04.584 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 3, 2024-09-16T04:08:04.584 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 8, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 5, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 1, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "is_stretch_pool": false, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 16, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 16, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 16, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 16, 2024-09-16T04:08:04.585 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 16, 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "29", 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-16T04:08:04.586 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-16T04:08:04.587 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "lrcprofile", 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-16T04:08:04.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 16384, 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "rados": {} 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.589 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4506422f-7ccf-46d7-af66-62e8cace167a", 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 17, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 27, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6824", 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6825", 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6826", 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6827", 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6830", 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6831", 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6828", 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6829", 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6825/3110165165", 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6827/3110165165", 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6831/3110165165", 2024-09-16T04:08:04.594 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6829/3110165165", 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "8dc0488e-efb6-4664-a799-5e14c56769b0", 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.595 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 27, 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3040423711 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3040423711 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3040423711 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.597 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3040423711 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3040423711 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.598 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3040423711 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3040423711 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2024-09-16T04:08:04.599 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3040423711 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6810/3040423711", 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6814/3040423711", 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6822/3040423711", 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6818/3040423711", 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "1686e657-c1f3-499b-8b10-67f428c5c216", 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 20, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.603 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/2266032822", 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/2266032822", 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/2266032822", 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/2266032822", 2024-09-16T04:08:04.605 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "730ad37c-1f5f-4d23-a1a7-c75320506665", 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.606 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 27, 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:08:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2024-09-16T04:08:04.608 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:08:04.609 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:08:04.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6811/3389562884", 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6815/3389562884", 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/3389562884", 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6819/3389562884", 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d82c5c44-72fe-4680-9c04-ab4e0775b929", 2024-09-16T04:08:04.611 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 27, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.612 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6810", 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6811", 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.613 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6812", 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6813", 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.614 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6816", 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6817", 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6814", 2024-09-16T04:08:04.615 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6815", 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.174:6811/1733996396", 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.174:6813/1733996396", 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.174:6817/1733996396", 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.174:6815/1733996396", 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "41502441-b0d9-460f-92c0-b8b3a915cac5", 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:08:04.617 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 27, 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6826", 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6827", 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.618 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6828", 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6829", 2024-09-16T04:08:04.619 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6832", 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6833", 2024-09-16T04:08:04.620 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6830", 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6831", 2024-09-16T04:08:04.621 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.174:6827/2956753716", 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.174:6829/2956753716", 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.174:6833/2956753716", 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.174:6831/2956753716", 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 6, 2024-09-16T04:08:04.622 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "45cc7521-11db-4496-8d0c-1abc42962810", 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 27, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6802", 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6803", 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6804", 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6805", 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6808", 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6809", 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.626 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6806", 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6807", 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.174:6803/2727832023", 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.174:6805/2727832023", 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.174:6809/2727832023", 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.174:6807/2727832023", 2024-09-16T04:08:04.627 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 7, 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "1b5d6337-73c3-494b-91cd-fcfa7d307050", 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 27, 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6818", 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6819", 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:08:04.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6820", 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6821", 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:08:04.630 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6824", 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6825", 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:08:04.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6822", 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6823", 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:08:04.632 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.174:6819/2338392542", 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.174:6821/2338392542", 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.174:6825/2338392542", 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.174:6823/2338392542", 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 8, 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "766efffa-3aff-4aa3-9cf8-88c5c7986d21", 2024-09-16T04:08:04.633 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 27, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6800", 2024-09-16T04:08:04.634 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6801", 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6802", 2024-09-16T04:08:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6803", 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6806", 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:08:04.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6807", 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6804", 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.638 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6805", 2024-09-16T04:08:04.638 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:08:04.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.638 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.194:6801/2621487148", 2024-09-16T04:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.194:6803/2621487148", 2024-09-16T04:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.194:6807/2621487148", 2024-09-16T04:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.194:6805/2621487148", 2024-09-16T04:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.640 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 9, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "755cc9aa-2f6f-4d7a-84cf-7feb74d65c48", 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.641 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6816", 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6817", 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6818", 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6819", 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.643 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6822", 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6823", 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.644 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6820", 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6821", 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.645 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.194:6817/367304498", 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.194:6819/367304498", 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.194:6823/367304498", 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.194:6821/367304498", 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 10, 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377", 2024-09-16T04:08:04.646 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.647 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6824", 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6825", 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6826", 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6827", 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6830", 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6831", 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.650 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6828", 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6829", 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.194:6825/3349503344", 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.194:6827/3349503344", 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.194:6831/3349503344", 2024-09-16T04:08:04.651 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.194:6829/3349503344", 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 11, 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "f71641eb-1331-4b39-9c0e-bf98e91ed295", 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:08:04.652 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 27, 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6808", 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.653 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6809", 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6810", 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6811", 2024-09-16T04:08:04.654 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6814", 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6815", 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:08:04.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6812", 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6813", 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:08:04.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.194:6809/2531896090", 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.194:6811/2531896090", 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.194:6815/2531896090", 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.194:6813/2531896090", 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:08:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.658 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:08:04.658 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2024-09-16T04:08:04.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.658 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-16T04:08:04.658 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:45.852767+0000", 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.659 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:45.938876+0000", 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:46.175090+0000", 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:45.893958+0000", 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.661 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:46.162300+0000", 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.662 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:47.123532+0000", 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 6, 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:46.160601+0000", 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 7, 2024-09-16T04:08:04.663 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:46.592432+0000", 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 8, 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.664 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:46.591765+0000", 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 9, 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:46.097182+0000", 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 10, 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:46.295219+0000", 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 11, 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:08:04.666 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:46.146281+0000", 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [], 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_primaries": [], 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2024-09-16T04:08:04.667 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "lrcprofile": { 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-device-class": "", 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-num-failure-domains": "0", 2024-09-16T04:08:04.668 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-osds-per-failure-domain": "0", 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-root": "default", 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "4", 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "l": "3", 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "2", 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "lrcprofile", 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "lrc" 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2024-09-16T04:08:04.669 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2024-09-16T04:08:04.670 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2024-09-16T04:08:04.670 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2024-09-16T04:08:04.670 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2024-09-16T04:08:04.670 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2024-09-16T04:08:04.670 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2024-09-16T04:08:04.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:04.670 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:08:04.671 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:04.671 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:04.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:08:04.674 INFO:tasks.thrashosds.thrasher:j is {'epoch': 29, 'fsid': '44381953-4509-4b15-853e-3887928dfa29', 'created': '2024-09-16T04:07:36.650572+0000', 'modified': '2024-09-16T04:08:04.062296+0000', 'last_up_change': '2024-09-16T04:07:48.919317+0000', 'last_in_change': '2024-09-16T04:07:39.793439+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T04:07:48.723300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T04:07:49.309634+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 23, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T04:08:01.082632+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 8, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'lrcprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '4506422f-7ccf-46d7-af66-62e8cace167a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6824', 'nonce': 3110165165}, {'type': 'v1', 'addr': '172.21.15.16:6825', 'nonce': 3110165165}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6826', 'nonce': 3110165165}, {'type': 'v1', 'addr': '172.21.15.16:6827', 'nonce': 3110165165}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6830', 'nonce': 3110165165}, {'type': 'v1', 'addr': '172.21.15.16:6831', 'nonce': 3110165165}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6828', 'nonce': 3110165165}, {'type': 'v1', 'addr': '172.21.15.16:6829', 'nonce': 3110165165}]}, 'public_addr': '172.21.15.16:6825/3110165165', 'cluster_addr': '172.21.15.16:6827/3110165165', 'heartbeat_back_addr': '172.21.15.16:6831/3110165165', 'heartbeat_front_addr': '172.21.15.16:6829/3110165165', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8dc0488e-efb6-4664-a799-5e14c56769b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 3040423711}, {'type': 'v1', 'addr': '172.21.15.16:6810', 'nonce': 3040423711}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 3040423711}, {'type': 'v1', 'addr': '172.21.15.16:6814', 'nonce': 3040423711}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 3040423711}, {'type': 'v1', 'addr': '172.21.15.16:6822', 'nonce': 3040423711}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 3040423711}, {'type': 'v1', 'addr': '172.21.15.16:6818', 'nonce': 3040423711}]}, 'public_addr': '172.21.15.16:6810/3040423711', 'cluster_addr': '172.21.15.16:6814/3040423711', 'heartbeat_back_addr': '172.21.15.16:6822/3040423711', 'heartbeat_front_addr': '172.21.15.16:6818/3040423711', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1686e657-c1f3-499b-8b10-67f428c5c216', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 2266032822}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 2266032822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 2266032822}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 2266032822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 2266032822}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 2266032822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 2266032822}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 2266032822}]}, 'public_addr': '172.21.15.16:6801/2266032822', 'cluster_addr': '172.21.15.16:6803/2266032822', 'heartbeat_back_addr': '172.21.15.16:6807/2266032822', 'heartbeat_front_addr': '172.21.15.16:6805/2266032822', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '730ad37c-1f5f-4d23-a1a7-c75320506665', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6809', 'nonce': 3389562884}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 3389562884}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6813', 'nonce': 3389562884}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 3389562884}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6821', 'nonce': 3389562884}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 3389562884}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6817', 'nonce': 3389562884}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 3389562884}]}, 'public_addr': '172.21.15.16:6811/3389562884', 'cluster_addr': '172.21.15.16:6815/3389562884', 'heartbeat_back_addr': '172.21.15.16:6823/3389562884', 'heartbeat_front_addr': '172.21.15.16:6819/3389562884', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd82c5c44-72fe-4680-9c04-ab4e0775b929', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6810', 'nonce': 1733996396}, {'type': 'v1', 'addr': '172.21.15.174:6811', 'nonce': 1733996396}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6812', 'nonce': 1733996396}, {'type': 'v1', 'addr': '172.21.15.174:6813', 'nonce': 1733996396}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6816', 'nonce': 1733996396}, {'type': 'v1', 'addr': '172.21.15.174:6817', 'nonce': 1733996396}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6814', 'nonce': 1733996396}, {'type': 'v1', 'addr': '172.21.15.174:6815', 'nonce': 1733996396}]}, 'public_addr': '172.21.15.174:6811/1733996396', 'cluster_addr': '172.21.15.174:6813/1733996396', 'heartbeat_back_addr': '172.21.15.174:6817/1733996396', 'heartbeat_front_addr': '172.21.15.174:6815/1733996396', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '41502441-b0d9-460f-92c0-b8b3a915cac5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6826', 'nonce': 2956753716}, {'type': 'v1', 'addr': '172.21.15.174:6827', 'nonce': 2956753716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6828', 'nonce': 2956753716}, {'type': 'v1', 'addr': '172.21.15.174:6829', 'nonce': 2956753716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6832', 'nonce': 2956753716}, {'type': 'v1', 'addr': '172.21.15.174:6833', 'nonce': 2956753716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6830', 'nonce': 2956753716}, {'type': 'v1', 'addr': '172.21.15.174:6831', 'nonce': 2956753716}]}, 'public_addr': '172.21.15.174:6827/2956753716', 'cluster_addr': '172.21.15.174:6829/2956753716', 'heartbeat_back_addr': '172.21.15.174:6833/2956753716', 'heartbeat_front_addr': '172.21.15.174:6831/2956753716', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45cc7521-11db-4496-8d0c-1abc42962810', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6802', 'nonce': 2727832023}, {'type': 'v1', 'addr': '172.21.15.174:6803', 'nonce': 2727832023}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6804', 'nonce': 2727832023}, {'type': 'v1', 'addr': '172.21.15.174:6805', 'nonce': 2727832023}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6808', 'nonce': 2727832023}, {'type': 'v1', 'addr': '172.21.15.174:6809', 'nonce': 2727832023}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6806', 'nonce': 2727832023}, {'type': 'v1', 'addr': '172.21.15.174:6807', 'nonce': 2727832023}]}, 'public_addr': '172.21.15.174:6803/2727832023', 'cluster_addr': '172.21.15.174:6805/2727832023', 'heartbeat_back_addr': '172.21.15.174:6809/2727832023', 'heartbeat_front_addr': '172.21.15.174:6807/2727832023', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b5d6337-73c3-494b-91cd-fcfa7d307050', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6818', 'nonce': 2338392542}, {'type': 'v1', 'addr': '172.21.15.174:6819', 'nonce': 2338392542}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6820', 'nonce': 2338392542}, {'type': 'v1', 'addr': '172.21.15.174:6821', 'nonce': 2338392542}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6824', 'nonce': 2338392542}, {'type': 'v1', 'addr': '172.21.15.174:6825', 'nonce': 2338392542}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6822', 'nonce': 2338392542}, {'type': 'v1', 'addr': '172.21.15.174:6823', 'nonce': 2338392542}]}, 'public_addr': '172.21.15.174:6819/2338392542', 'cluster_addr': '172.21.15.174:6821/2338392542', 'heartbeat_back_addr': '172.21.15.174:6825/2338392542', 'heartbeat_front_addr': '172.21.15.174:6823/2338392542', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '766efffa-3aff-4aa3-9cf8-88c5c7986d21', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6800', 'nonce': 2621487148}, {'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 2621487148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6802', 'nonce': 2621487148}, {'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 2621487148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6806', 'nonce': 2621487148}, {'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 2621487148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6804', 'nonce': 2621487148}, {'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 2621487148}]}, 'public_addr': '172.21.15.194:6801/2621487148', 'cluster_addr': '172.21.15.194:6803/2621487148', 'heartbeat_back_addr': '172.21.15.194:6807/2621487148', 'heartbeat_front_addr': '172.21.15.194:6805/2621487148', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '755cc9aa-2f6f-4d7a-84cf-7feb74d65c48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6816', 'nonce': 367304498}, {'type': 'v1', 'addr': '172.21.15.194:6817', 'nonce': 367304498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6818', 'nonce': 367304498}, {'type': 'v1', 'addr': '172.21.15.194:6819', 'nonce': 367304498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6822', 'nonce': 367304498}, {'type': 'v1', 'addr': '172.21.15.194:6823', 'nonce': 367304498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6820', 'nonce': 367304498}, {'type': 'v1', 'addr': '172.21.15.194:6821', 'nonce': 367304498}]}, 'public_addr': '172.21.15.194:6817/367304498', 'cluster_addr': '172.21.15.194:6819/367304498', 'heartbeat_back_addr': '172.21.15.194:6823/367304498', 'heartbeat_front_addr': '172.21.15.194:6821/367304498', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6824', 'nonce': 3349503344}, {'type': 'v1', 'addr': '172.21.15.194:6825', 'nonce': 3349503344}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6826', 'nonce': 3349503344}, {'type': 'v1', 'addr': '172.21.15.194:6827', 'nonce': 3349503344}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6830', 'nonce': 3349503344}, {'type': 'v1', 'addr': '172.21.15.194:6831', 'nonce': 3349503344}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6828', 'nonce': 3349503344}, {'type': 'v1', 'addr': '172.21.15.194:6829', 'nonce': 3349503344}]}, 'public_addr': '172.21.15.194:6825/3349503344', 'cluster_addr': '172.21.15.194:6827/3349503344', 'heartbeat_back_addr': '172.21.15.194:6831/3349503344', 'heartbeat_front_addr': '172.21.15.194:6829/3349503344', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f71641eb-1331-4b39-9c0e-bf98e91ed295', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6808', 'nonce': 2531896090}, {'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 2531896090}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6810', 'nonce': 2531896090}, {'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 2531896090}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6814', 'nonce': 2531896090}, {'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 2531896090}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6812', 'nonce': 2531896090}, {'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 2531896090}]}, 'public_addr': '172.21.15.194:6809/2531896090', 'cluster_addr': '172.21.15.194:6811/2531896090', 'heartbeat_back_addr': '172.21.15.194:6815/2531896090', 'heartbeat_front_addr': '172.21.15.194:6813/2531896090', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:45.852767+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:45.938876+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:46.175090+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:45.893958+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:46.162300+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:47.123532+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:46.160601+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:46.592432+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:46.591765+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:46.097182+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:46.295219+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:46.146281+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'lrcprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'k': '4', 'l': '3', 'm': '2', 'name': 'lrcprofile', 'plugin': 'lrc'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T04:08:04.674 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:08:04.674 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:04.669+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:04.698 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:04.698 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:04.721 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:04.775 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:04.773+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:04.807 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:04.807 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:04.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:08:04.901 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:04.899+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:04.965 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:04.965 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:04.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:04.975 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:04.973+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:04.998 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:08:04.998 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:08:05.012 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-09-16T04:08:04.670816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":44,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1132462080,"kb_used":340912,"kb_used_data":4072,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1132121168,"statfs":{"total":1159641169920,"available":1159292076032,"internally_reserved":0,"allocated":4169728,"data_stored":2363937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":344634684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000715"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.224679+0000","last_change":"2024-09-16T04:08:03.222197+0000","last_active":"2024-09-16T04:08:03.224679+0000","last_peered":"2024-09-16T04:08:03.224679+0000","last_clean":"2024-09-16T04:08:03.224679+0000","last_became_active":"2024-09-16T04:08:03.221547+0000","last_became_peered":"2024-09-16T04:08:03.221547+0000","last_unstale":"2024-09-16T04:08:03.224679+0000","last_undegraded":"2024-09-16T04:08:03.224679+0000","last_fullsized":"2024-09-16T04:08:03.224679+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:03.187613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8,2,9,10,0,4],"acting":[7,11,8,2,9,10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.223209+0000","last_change":"2024-09-16T04:08:03.221516+0000","last_active":"2024-09-16T04:08:03.223209+0000","last_peered":"2024-09-16T04:08:03.223209+0000","last_clean":"2024-09-16T04:08:03.223209+0000","last_became_active":"2024-09-16T04:08:03.220935+0000","last_became_peered":"2024-09-16T04:08:03.220935+0000","last_unstale":"2024-09-16T04:08:03.223209+0000","last_undegraded":"2024-09-16T04:08:03.223209+0000","last_fullsized":"2024-09-16T04:08:03.223209+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:12.032786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,4,0,9,3,10],"acting":[11,8,1,4,0,9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.212089+0000","last_change":"2024-09-16T04:08:03.209161+0000","last_active":"2024-09-16T04:08:03.212089+0000","last_peered":"2024-09-16T04:08:03.212089+0000","last_clean":"2024-09-16T04:08:03.212089+0000","last_became_active":"2024-09-16T04:08:03.208404+0000","last_became_peered":"2024-09-16T04:08:03.208404+0000","last_unstale":"2024-09-16T04:08:03.212089+0000","last_undegraded":"2024-09-16T04:08:03.212089+0000","last_fullsized":"2024-09-16T04:08:03.212089+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:29.986069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9,6,11,3,4,0],"acting":[7,5,9,6,11,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:08:02.050701+0000","last_change":"2024-09-16T04:08:02.050701+0000","last_active":"2024-09-16T04:08:02.050701+0000","last_peered":"2024-09-16T04:08:02.050701+0000","last_clean":"2024-09-16T04:08:02.050701+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:08:02.050701+0000","last_undegraded":"2024-09-16T04:08:02.050701+0000","last_fullsized":"2024-09-16T04:08:02.050701+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:08:02.050701+0000","last_change":"2024-09-16T04:08:02.050701+0000","last_active":"2024-09-16T04:08:02.050701+0000","last_peered":"2024-09-16T04:08:02.050701+0000","last_clean":"2024-09-16T04:08:02.050701+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:08:02.050701+0000","last_undegraded":"2024-09-16T04:08:02.050701+0000","last_fullsized":"2024-09-16T04:08:02.050701+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:08:02.050701+0000","last_change":"2024-09-16T04:08:02.050701+0000","last_active":"2024-09-16T04:08:02.050701+0000","last_peered":"2024-09-16T04:08:02.050701+0000","last_clean":"2024-09-16T04:08:02.050701+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:08:02.050701+0000","last_undegraded":"2024-09-16T04:08:02.050701+0000","last_fullsized":"2024-09-16T04:08:02.050701+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.222590+0000","last_change":"2024-09-16T04:08:03.220063+0000","last_active":"2024-09-16T04:08:03.222590+0000","last_peered":"2024-09-16T04:08:03.222590+0000","last_clean":"2024-09-16T04:08:03.222590+0000","last_became_active":"2024-09-16T04:08:03.219327+0000","last_became_peered":"2024-09-16T04:08:03.219327+0000","last_unstale":"2024-09-16T04:08:03.222590+0000","last_undegraded":"2024-09-16T04:08:03.222590+0000","last_fullsized":"2024-09-16T04:08:03.222590+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:29.304954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,9,5,1,4,10],"acting":[8,11,7,9,5,1,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.219653+0000","last_change":"2024-09-16T04:08:03.219043+0000","last_active":"2024-09-16T04:08:03.219653+0000","last_peered":"2024-09-16T04:08:03.219653+0000","last_clean":"2024-09-16T04:08:03.219653+0000","last_became_active":"2024-09-16T04:08:03.218478+0000","last_became_peered":"2024-09-16T04:08:03.218478+0000","last_unstale":"2024-09-16T04:08:03.219653+0000","last_undegraded":"2024-09-16T04:08:03.219653+0000","last_fullsized":"2024-09-16T04:08:03.219653+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:27.982118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,4,3,5,10,8],"acting":[11,1,2,4,3,5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":199,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.076781+0000","last_change":"2024-09-16T04:07:50.119798+0000","last_active":"2024-09-16T04:08:03.076781+0000","last_peered":"2024-09-16T04:08:03.076781+0000","last_clean":"2024-09-16T04:08:03.076781+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:08:03.076781+0000","last_undegraded":"2024-09-16T04:08:03.076781+0000","last_fullsized":"2024-09-16T04:08:03.076781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:07:48.919317+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:53.937788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T04:07:55.426549+0000","last_change":"2024-09-16T04:07:52.943663+0000","last_active":"2024-09-16T04:07:55.426549+0000","last_peered":"2024-09-16T04:07:55.426549+0000","last_clean":"2024-09-16T04:07:55.426549+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:07:55.426549+0000","last_undegraded":"2024-09-16T04:07:55.426549+0000","last_fullsized":"2024-09-16T04:07:55.426549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.855553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:08:02.050701+0000","last_change":"2024-09-16T04:08:02.050701+0000","last_active":"2024-09-16T04:08:02.050701+0000","last_peered":"2024-09-16T04:08:02.050701+0000","last_clean":"2024-09-16T04:08:02.050701+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:08:02.050701+0000","last_undegraded":"2024-09-16T04:08:02.050701+0000","last_fullsized":"2024-09-16T04:08:02.050701+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.076124+0000","last_change":"2024-09-16T04:07:52.942253+0000","last_active":"2024-09-16T04:08:03.076124+0000","last_peered":"2024-09-16T04:08:03.076124+0000","last_clean":"2024-09-16T04:08:03.076124+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:08:03.076124+0000","last_undegraded":"2024-09-16T04:08:03.076124+0000","last_fullsized":"2024-09-16T04:08:03.076124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.104501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:08:02.050701+0000","last_change":"2024-09-16T04:08:02.050701+0000","last_active":"2024-09-16T04:08:02.050701+0000","last_peered":"2024-09-16T04:08:02.050701+0000","last_clean":"2024-09-16T04:08:02.050701+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:08:02.050701+0000","last_undegraded":"2024-09-16T04:08:02.050701+0000","last_fullsized":"2024-09-16T04:08:02.050701+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T04:07:56.130234+0000","last_change":"2024-09-16T04:07:53.018517+0000","last_active":"2024-09-16T04:07:56.130234+0000","last_peered":"2024-09-16T04:07:56.130234+0000","last_clean":"2024-09-16T04:07:56.130234+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:07:56.130234+0000","last_undegraded":"2024-09-16T04:07:56.130234+0000","last_fullsized":"2024-09-16T04:07:56.130234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.209603+0000","last_change":"2024-09-16T04:08:03.208669+0000","last_active":"2024-09-16T04:08:03.209603+0000","last_peered":"2024-09-16T04:08:03.209603+0000","last_clean":"2024-09-16T04:08:03.209603+0000","last_became_active":"2024-09-16T04:08:03.208071+0000","last_became_peered":"2024-09-16T04:08:03.208071+0000","last_unstale":"2024-09-16T04:08:03.209603+0000","last_undegraded":"2024-09-16T04:08:03.209603+0000","last_fullsized":"2024-09-16T04:08:03.209603+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:02.998545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6,1,7,8,0,5],"acting":[11,9,6,1,7,8,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T04:07:55.426509+0000","last_change":"2024-09-16T04:07:52.944002+0000","last_active":"2024-09-16T04:07:55.426509+0000","last_peered":"2024-09-16T04:07:55.426509+0000","last_clean":"2024-09-16T04:07:55.426509+0000","last_became_active":"2024-09-16T04:07:50.935310+0000","last_became_peered":"2024-09-16T04:07:50.935310+0000","last_unstale":"2024-09-16T04:07:55.426509+0000","last_undegraded":"2024-09-16T04:07:55.426509+0000","last_fullsized":"2024-09-16T04:07:55.426509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:03.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:08:02.050701+0000","last_change":"2024-09-16T04:08:02.050701+0000","last_active":"2024-09-16T04:08:02.050701+0000","last_peered":"2024-09-16T04:08:02.050701+0000","last_clean":"2024-09-16T04:08:02.050701+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:08:02.050701+0000","last_undegraded":"2024-09-16T04:08:02.050701+0000","last_fullsized":"2024-09-16T04:08:02.050701+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:08:02.050701+0000","last_change":"2024-09-16T04:08:02.050701+0000","last_active":"2024-09-16T04:08:02.050701+0000","last_peered":"2024-09-16T04:08:02.050701+0000","last_clean":"2024-09-16T04:08:02.050701+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:08:02.050701+0000","last_undegraded":"2024-09-16T04:08:02.050701+0000","last_fullsized":"2024-09-16T04:08:02.050701+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.069542+0000","last_change":"2024-09-16T04:07:52.946320+0000","last_active":"2024-09-16T04:08:03.069542+0000","last_peered":"2024-09-16T04:08:03.069542+0000","last_clean":"2024-09-16T04:08:03.069542+0000","last_became_active":"2024-09-16T04:07:50.936587+0000","last_became_peered":"2024-09-16T04:07:50.936587+0000","last_unstale":"2024-09-16T04:08:03.069542+0000","last_undegraded":"2024-09-16T04:08:03.069542+0000","last_fullsized":"2024-09-16T04:08:03.069542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:13.247770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051151899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.225219+0000","last_change":"2024-09-16T04:08:03.221041+0000","last_active":"2024-09-16T04:08:03.225219+0000","last_peered":"2024-09-16T04:08:03.225219+0000","last_clean":"2024-09-16T04:08:03.225219+0000","last_became_active":"2024-09-16T04:08:03.219952+0000","last_became_peered":"2024-09-16T04:08:03.219952+0000","last_unstale":"2024-09-16T04:08:03.225219+0000","last_undegraded":"2024-09-16T04:08:03.225219+0000","last_fullsized":"2024-09-16T04:08:03.225219+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:21.955994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,9,5,10,2,6,0],"acting":[1,11,9,5,10,2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.069344+0000","last_change":"2024-09-16T04:07:52.939889+0000","last_active":"2024-09-16T04:08:03.069344+0000","last_peered":"2024-09-16T04:08:03.069344+0000","last_clean":"2024-09-16T04:08:03.069344+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:08:03.069344+0000","last_undegraded":"2024-09-16T04:08:03.069344+0000","last_fullsized":"2024-09-16T04:08:03.069344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:16.774685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:08:02.050701+0000","last_change":"2024-09-16T04:08:02.050701+0000","last_active":"2024-09-16T04:08:02.050701+0000","last_peered":"2024-09-16T04:08:02.050701+0000","last_clean":"2024-09-16T04:08:02.050701+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:08:02.050701+0000","last_undegraded":"2024-09-16T04:08:02.050701+0000","last_fullsized":"2024-09-16T04:08:02.050701+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T04:08:03.069733+0000","last_change":"2024-09-16T04:07:52.946321+0000","last_active":"2024-09-16T04:08:03.069733+0000","last_peered":"2024-09-16T04:08:03.069733+0000","last_clean":"2024-09-16T04:08:03.069733+0000","last_became_active":"2024-09-16T04:07:50.937245+0000","last_became_peered":"2024-09-16T04:07:50.937245+0000","last_unstale":"2024-09-16T04:08:03.069733+0000","last_undegraded":"2024-09-16T04:08:03.069733+0000","last_fullsized":"2024-09-16T04:08:03.069733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:18.129176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051620099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T04:08:02.050701+0000","last_change":"2024-09-16T04:08:02.050701+0000","last_active":"2024-09-16T04:08:02.050701+0000","last_peered":"2024-09-16T04:08:02.050701+0000","last_clean":"2024-09-16T04:08:02.050701+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T04:08:02.050701+0000","last_undegraded":"2024-09-16T04:08:02.050701+0000","last_fullsized":"2024-09-16T04:08:02.050701+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T04:07:55.425993+0000","last_change":"2024-09-16T04:07:52.941363+0000","last_active":"2024-09-16T04:07:55.425993+0000","last_peered":"2024-09-16T04:07:55.425993+0000","last_clean":"2024-09-16T04:07:55.425993+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:07:55.425993+0000","last_undegraded":"2024-09-16T04:07:55.425993+0000","last_fullsized":"2024-09-16T04:07:55.425993+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:59.932581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":56,"acting":56,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378629,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":92229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":92248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":92229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378631,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":92229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378629,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":92229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411333,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":815104,"data_stored":654481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":125553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411333,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":117172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411333,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":125553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":815104,"data_stored":654481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411334,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":125572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444038,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":245760,"data_stored":99961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:08:05.013 INFO:tasks.thrashosds.thrasher:Setting 3.d to [7, 8, 4, 11, 9, 0, 1, 5] 2024-09-16T04:08:05.013 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.d', '7', '8', '4', '11', '9', '0', '1', '5'] 2024-09-16T04:08:05.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.d 7 8 4 11 9 0 1 5 2024-09-16T04:08:05.074 INFO:tasks.rados.rados.0.smithi016.stdout:1: write initial oid 1 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:1: seq_num 0 ranges {0=32768} 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:1: writing smithi01639990-1 from 0 to 32768 tid 1 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:2: write initial oid 2 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:2: seq_num 1 ranges {0=32768} 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:2: writing smithi01639990-2 from 0 to 32768 tid 1 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:3: write initial oid 3 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:3: seq_num 2 ranges {0=49152} 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:3: writing smithi01639990-3 from 0 to 49152 tid 1 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:4: write initial oid 4 2024-09-16T04:08:05.075 INFO:tasks.rados.rados.0.smithi016.stdout:4: seq_num 3 ranges {0=65536} 2024-09-16T04:08:05.076 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:05.075+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:4: writing smithi01639990-4 from 0 to 65536 tid 1 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:5: write initial oid 5 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:5: seq_num 4 ranges {0=81920} 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:5: writing smithi01639990-5 from 0 to 81920 tid 1 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:6: write initial oid 6 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:6: seq_num 5 ranges {0=98304} 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:6: writing smithi01639990-6 from 0 to 98304 tid 1 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:7: write initial oid 7 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:7: seq_num 6 ranges {0=114688} 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:7: writing smithi01639990-7 from 0 to 114688 tid 1 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:8: write initial oid 8 2024-09-16T04:08:05.078 INFO:tasks.rados.rados.0.smithi016.stdout:8: seq_num 7 ranges {0=131072} 2024-09-16T04:08:05.079 INFO:tasks.rados.rados.0.smithi016.stdout:8: writing smithi01639990-8 from 0 to 131072 tid 1 2024-09-16T04:08:05.079 INFO:tasks.rados.rados.0.smithi016.stdout:9: write initial oid 9 2024-09-16T04:08:05.079 INFO:tasks.rados.rados.0.smithi016.stdout:9: seq_num 8 ranges {0=147456} 2024-09-16T04:08:05.079 INFO:tasks.rados.rados.0.smithi016.stdout:9: writing smithi01639990-9 from 0 to 147456 tid 1 2024-09-16T04:08:05.080 INFO:tasks.rados.rados.0.smithi016.stdout:10: write initial oid 10 2024-09-16T04:08:05.080 INFO:tasks.rados.rados.0.smithi016.stdout:10: seq_num 9 ranges {0=163840} 2024-09-16T04:08:05.080 INFO:tasks.rados.rados.0.smithi016.stdout:10: writing smithi01639990-10 from 0 to 163840 tid 1 2024-09-16T04:08:05.080 INFO:tasks.rados.rados.0.smithi016.stdout:11: write initial oid 11 2024-09-16T04:08:05.080 INFO:tasks.rados.rados.0.smithi016.stdout:11: seq_num 10 ranges {0=180224} 2024-09-16T04:08:05.082 INFO:tasks.rados.rados.0.smithi016.stdout:11: writing smithi01639990-11 from 0 to 180224 tid 1 2024-09-16T04:08:05.082 INFO:tasks.rados.rados.0.smithi016.stdout:12: write initial oid 12 2024-09-16T04:08:05.082 INFO:tasks.rados.rados.0.smithi016.stdout:12: seq_num 11 ranges {0=196608} 2024-09-16T04:08:05.084 INFO:tasks.rados.rados.0.smithi016.stdout:12: writing smithi01639990-12 from 0 to 196608 tid 1 2024-09-16T04:08:05.084 INFO:tasks.rados.rados.0.smithi016.stdout:13: write initial oid 13 2024-09-16T04:08:05.084 INFO:tasks.rados.rados.0.smithi016.stdout:13: seq_num 12 ranges {0=212992} 2024-09-16T04:08:05.088 INFO:tasks.rados.rados.0.smithi016.stdout:13: writing smithi01639990-13 from 0 to 212992 tid 1 2024-09-16T04:08:05.088 INFO:tasks.rados.rados.0.smithi016.stdout:14: write initial oid 14 2024-09-16T04:08:05.088 INFO:tasks.rados.rados.0.smithi016.stdout:14: seq_num 13 ranges {0=229376} 2024-09-16T04:08:05.088 INFO:tasks.rados.rados.0.smithi016.stdout:14: writing smithi01639990-14 from 0 to 229376 tid 1 2024-09-16T04:08:05.088 INFO:tasks.rados.rados.0.smithi016.stdout:15: write initial oid 15 2024-09-16T04:08:05.088 INFO:tasks.rados.rados.0.smithi016.stdout:15: seq_num 14 ranges {0=245760} 2024-09-16T04:08:05.089 INFO:tasks.rados.rados.0.smithi016.stdout:15: writing smithi01639990-15 from 0 to 245760 tid 1 2024-09-16T04:08:05.089 INFO:tasks.rados.rados.0.smithi016.stdout:16: write initial oid 16 2024-09-16T04:08:05.090 INFO:tasks.rados.rados.0.smithi016.stdout:16: seq_num 15 ranges {0=262144} 2024-09-16T04:08:05.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:08:05.094 INFO:tasks.rados.rados.0.smithi016.stdout:16: writing smithi01639990-16 from 0 to 262144 tid 1 2024-09-16T04:08:05.094 INFO:tasks.rados.rados.0.smithi016.stdout:3: finishing write tid 1 to smithi01639990-3 2024-09-16T04:08:05.094 INFO:tasks.rados.rados.0.smithi016.stdout:3: finishing write tid 2 to smithi01639990-3 2024-09-16T04:08:05.094 INFO:tasks.rados.rados.0.smithi016.stdout:1: finishing write tid 1 to smithi01639990-1 2024-09-16T04:08:05.094 INFO:tasks.rados.rados.0.smithi016.stdout:1: finishing write tid 2 to smithi01639990-1 2024-09-16T04:08:05.094 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 16 2024-09-16T04:08:05.097 INFO:tasks.rados.rados.0.smithi016.stdout:4: finishing write tid 1 to smithi01639990-4 2024-09-16T04:08:05.097 INFO:tasks.rados.rados.0.smithi016.stdout:4: finishing write tid 2 to smithi01639990-4 2024-09-16T04:08:05.097 INFO:tasks.rados.rados.0.smithi016.stdout:2: finishing write tid 1 to smithi01639990-2 2024-09-16T04:08:05.099 INFO:tasks.rados.rados.0.smithi016.stdout:4: finishing write tid 3 to smithi01639990-4 2024-09-16T04:08:05.100 INFO:tasks.rados.rados.0.smithi016.stdout:4: oid 4 updating version 0 to 1 2024-09-16T04:08:05.100 INFO:tasks.rados.rados.0.smithi016.stdout:4: oid 4 updating version 1 to 2 2024-09-16T04:08:05.100 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T04:08:05.100 INFO:tasks.rados.rados.0.smithi016.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-16T04:08:05.101 INFO:tasks.rados.rados.0.smithi016.stdout:2: finishing write tid 2 to smithi01639990-2 2024-09-16T04:08:05.102 INFO:tasks.rados.rados.0.smithi016.stdout:4: done (15 left) 2024-09-16T04:08:05.102 INFO:tasks.rados.rados.0.smithi016.stdout:17: write initial oid 17 2024-09-16T04:08:05.102 INFO:tasks.rados.rados.0.smithi016.stdout:17: seq_num 16 ranges {0=278528} 2024-09-16T04:08:05.104 INFO:tasks.rados.rados.0.smithi016.stdout:17: writing smithi01639990-17 from 0 to 278528 tid 1 2024-09-16T04:08:05.104 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 16 2024-09-16T04:08:05.104 INFO:tasks.rados.rados.0.smithi016.stdout:9: finishing write tid 1 to smithi01639990-9 2024-09-16T04:08:05.104 INFO:tasks.rados.rados.0.smithi016.stdout:11: finishing write tid 1 to smithi01639990-11 2024-09-16T04:08:05.104 INFO:tasks.rados.rados.0.smithi016.stdout:11: finishing write tid 2 to smithi01639990-11 2024-09-16T04:08:05.104 INFO:tasks.rados.rados.0.smithi016.stdout:3: finishing write tid 3 to smithi01639990-3 2024-09-16T04:08:05.105 INFO:tasks.rados.rados.0.smithi016.stdout:3: oid 3 updating version 0 to 1 2024-09-16T04:08:05.105 INFO:tasks.rados.rados.0.smithi016.stdout:3: oid 3 updating version 1 to 2 2024-09-16T04:08:05.105 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T04:08:05.105 INFO:tasks.rados.rados.0.smithi016.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-16T04:08:05.105 INFO:tasks.rados.rados.0.smithi016.stdout:5: finishing write tid 1 to smithi01639990-5 2024-09-16T04:08:05.105 INFO:tasks.rados.rados.0.smithi016.stdout:5: finishing write tid 2 to smithi01639990-5 2024-09-16T04:08:05.105 INFO:tasks.rados.rados.0.smithi016.stdout:10: finishing write tid 1 to smithi01639990-10 2024-09-16T04:08:05.105 INFO:tasks.rados.rados.0.smithi016.stdout:10: finishing write tid 2 to smithi01639990-10 2024-09-16T04:08:05.105 INFO:tasks.rados.rados.0.smithi016.stdout:6: finishing write tid 1 to smithi01639990-6 2024-09-16T04:08:05.108 INFO:tasks.rados.rados.0.smithi016.stdout:3: done (15 left) 2024-09-16T04:08:05.108 INFO:tasks.rados.rados.0.smithi016.stdout:18: write initial oid 18 2024-09-16T04:08:05.108 INFO:tasks.rados.rados.0.smithi016.stdout:18: seq_num 17 ranges {0=294912} 2024-09-16T04:08:05.108 INFO:tasks.rados.rados.0.smithi016.stdout:18: writing smithi01639990-18 from 0 to 294912 tid 1 2024-09-16T04:08:05.108 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 16 2024-09-16T04:08:05.108 INFO:tasks.rados.rados.0.smithi016.stdout:6: finishing write tid 2 to smithi01639990-6 2024-09-16T04:08:05.108 INFO:tasks.rados.rados.0.smithi016.stdout:9: finishing write tid 2 to smithi01639990-9 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:11: finishing write tid 3 to smithi01639990-11 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:11: oid 11 updating version 0 to 1 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:11: oid 11 updating version 1 to 2 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:10: finishing write tid 3 to smithi01639990-10 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:10: oid 10 updating version 0 to 2 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:10: oid 10 version 2 is already newer than 1 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:10: done (15 left) 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:11: done (14 left) 2024-09-16T04:08:05.109 INFO:tasks.rados.rados.0.smithi016.stdout:19: write initial oid 19 2024-09-16T04:08:05.110 INFO:tasks.rados.rados.0.smithi016.stdout:19: seq_num 18 ranges {0=311296} 2024-09-16T04:08:05.110 INFO:tasks.rados.rados.0.smithi016.stdout:19: writing smithi01639990-19 from 0 to 311296 tid 1 2024-09-16T04:08:05.110 INFO:tasks.rados.rados.0.smithi016.stdout:20: write initial oid 20 2024-09-16T04:08:05.110 INFO:tasks.rados.rados.0.smithi016.stdout:20: seq_num 19 ranges {0=327680} 2024-09-16T04:08:05.116 INFO:tasks.rados.rados.0.smithi016.stdout:20: writing smithi01639990-20 from 0 to 327680 tid 1 2024-09-16T04:08:05.116 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 16 2024-09-16T04:08:05.116 INFO:tasks.rados.rados.0.smithi016.stdout:8: finishing write tid 1 to smithi01639990-8 2024-09-16T04:08:05.116 INFO:tasks.rados.rados.0.smithi016.stdout:8: finishing write tid 2 to smithi01639990-8 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:14: finishing write tid 1 to smithi01639990-14 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:14: finishing write tid 2 to smithi01639990-14 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:8: finishing write tid 3 to smithi01639990-8 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:8: oid 8 updating version 0 to 1 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:8: oid 8 updating version 1 to 2 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:16: finishing write tid 1 to smithi01639990-16 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:13: finishing write tid 1 to smithi01639990-13 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:13: finishing write tid 2 to smithi01639990-13 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:16: finishing write tid 2 to smithi01639990-16 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:1: finishing write tid 3 to smithi01639990-1 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:1: oid 1 updating version 0 to 1 2024-09-16T04:08:05.117 INFO:tasks.rados.rados.0.smithi016.stdout:1: oid 1 updating version 1 to 2 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:2: finishing write tid 3 to smithi01639990-2 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:2: oid 2 updating version 0 to 1 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:2: oid 2 updating version 1 to 2 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:7: finishing write tid 1 to smithi01639990-7 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:12: finishing write tid 1 to smithi01639990-12 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:12: finishing write tid 2 to smithi01639990-12 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:14: finishing write tid 3 to smithi01639990-14 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:14: oid 14 updating version 0 to 4 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:14: oid 14 version 4 is already newer than 3 2024-09-16T04:08:05.118 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:6: finishing write tid 3 to smithi01639990-6 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:6: oid 6 updating version 0 to 1 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:6: oid 6 updating version 1 to 2 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:5: finishing write tid 3 to smithi01639990-5 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:5: oid 5 updating version 0 to 1 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:5: oid 5 updating version 1 to 2 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:13: finishing write tid 3 to smithi01639990-13 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:13: oid 13 updating version 0 to 4 2024-09-16T04:08:05.119 INFO:tasks.rados.rados.0.smithi016.stdout:13: oid 13 version 4 is already newer than 3 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:1: done (15 left) 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:2: done (14 left) 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:5: done (13 left) 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:6: done (12 left) 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:8: done (11 left) 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:13: done (10 left) 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:14: done (9 left) 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:21: write initial oid 21 2024-09-16T04:08:05.120 INFO:tasks.rados.rados.0.smithi016.stdout:21: seq_num 20 ranges {0=344064} 2024-09-16T04:08:05.121 INFO:tasks.rados.rados.0.smithi016.stdout:21: writing smithi01639990-21 from 0 to 344064 tid 1 2024-09-16T04:08:05.121 INFO:tasks.rados.rados.0.smithi016.stdout:16: finishing write tid 3 to smithi01639990-16 2024-09-16T04:08:05.121 INFO:tasks.rados.rados.0.smithi016.stdout:16: oid 16 updating version 0 to 4 2024-09-16T04:08:05.121 INFO:tasks.rados.rados.0.smithi016.stdout:16: oid 16 version 4 is already newer than 3 2024-09-16T04:08:05.121 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 16 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T04:08:05.121 INFO:tasks.rados.rados.0.smithi016.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-16T04:08:05.121 INFO:tasks.rados.rados.0.smithi016.stdout:15: finishing write tid 1 to smithi01639990-15 2024-09-16T04:08:05.121 INFO:tasks.rados.rados.0.smithi016.stdout:15: finishing write tid 2 to smithi01639990-15 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:9: finishing write tid 3 to smithi01639990-9 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:9: oid 9 updating version 0 to 1 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:9: oid 9 updating version 1 to 2 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:17: finishing write tid 1 to smithi01639990-17 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:17: finishing write tid 2 to smithi01639990-17 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:12: finishing write tid 3 to smithi01639990-12 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:12: oid 12 updating version 0 to 1 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:12: oid 12 updating version 1 to 2 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:08:05.122 INFO:tasks.rados.rados.0.smithi016.stdout:9: done (9 left) 2024-09-16T04:08:05.123 INFO:tasks.rados.rados.0.smithi016.stdout:12: done (8 left) 2024-09-16T04:08:05.123 INFO:tasks.rados.rados.0.smithi016.stdout:16: done (7 left) 2024-09-16T04:08:05.123 INFO:tasks.rados.rados.0.smithi016.stdout:22: write initial oid 22 2024-09-16T04:08:05.123 INFO:tasks.rados.rados.0.smithi016.stdout:22: seq_num 21 ranges {0=360448} 2024-09-16T04:08:05.123 INFO:tasks.rados.rados.0.smithi016.stdout:22: writing smithi01639990-22 from 0 to 360448 tid 1 2024-09-16T04:08:05.123 INFO:tasks.rados.rados.0.smithi016.stdout:23: write initial oid 23 2024-09-16T04:08:05.123 INFO:tasks.rados.rados.0.smithi016.stdout:23: seq_num 22 ranges {0=376832} 2024-09-16T04:08:05.124 INFO:tasks.rados.rados.0.smithi016.stdout:23: writing smithi01639990-23 from 0 to 376832 tid 1 2024-09-16T04:08:05.124 INFO:tasks.rados.rados.0.smithi016.stdout:24: write initial oid 24 2024-09-16T04:08:05.124 INFO:tasks.rados.rados.0.smithi016.stdout:24: seq_num 23 ranges {0=393216} 2024-09-16T04:08:05.128 INFO:tasks.rados.rados.0.smithi016.stdout:24: writing smithi01639990-24 from 0 to 393216 tid 1 2024-09-16T04:08:05.128 INFO:tasks.rados.rados.0.smithi016.stdout:25: write initial oid 25 2024-09-16T04:08:05.128 INFO:tasks.rados.rados.0.smithi016.stdout:25: seq_num 24 ranges {0=409600} 2024-09-16T04:08:05.131 INFO:tasks.rados.rados.0.smithi016.stdout:25: writing smithi01639990-25 from 0 to 409600 tid 1 2024-09-16T04:08:05.131 INFO:tasks.rados.rados.0.smithi016.stdout:7: finishing write tid 2 to smithi01639990-7 2024-09-16T04:08:05.131 INFO:tasks.rados.rados.0.smithi016.stdout:19: finishing write tid 1 to smithi01639990-19 2024-09-16T04:08:05.131 INFO:tasks.rados.rados.0.smithi016.stdout:20: finishing write tid 1 to smithi01639990-20 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:7: finishing write tid 3 to smithi01639990-7 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:7: oid 7 updating version 0 to 1 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:7: oid 7 updating version 1 to 2 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:17: finishing write tid 3 to smithi01639990-17 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:17: oid 17 updating version 0 to 2 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:17: oid 17 version 2 is already newer than 1 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:19: finishing write tid 2 to smithi01639990-19 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:20: finishing write tid 2 to smithi01639990-20 2024-09-16T04:08:05.132 INFO:tasks.rados.rados.0.smithi016.stdout:15: finishing write tid 3 to smithi01639990-15 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:15: oid 15 updating version 0 to 4 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:15: oid 15 version 4 is already newer than 3 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:19: finishing write tid 3 to smithi01639990-19 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:19: oid 19 updating version 0 to 3 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:19: oid 19 updating version 3 to 4 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 19 v 4 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:18: finishing write tid 1 to smithi01639990-18 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:7: done (10 left) 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:15: done (9 left) 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:17: done (8 left) 2024-09-16T04:08:05.133 INFO:tasks.rados.rados.0.smithi016.stdout:19: done (7 left) 2024-09-16T04:08:05.134 INFO:tasks.rados.rados.0.smithi016.stdout:26: write initial oid 26 2024-09-16T04:08:05.134 INFO:tasks.rados.rados.0.smithi016.stdout:26: seq_num 25 ranges {0=425984} 2024-09-16T04:08:05.134 INFO:tasks.rados.rados.0.smithi016.stdout:26: writing smithi01639990-26 from 0 to 425984 tid 1 2024-09-16T04:08:05.134 INFO:tasks.rados.rados.0.smithi016.stdout:27: write initial oid 27 2024-09-16T04:08:05.134 INFO:tasks.rados.rados.0.smithi016.stdout:27: seq_num 26 ranges {0=442368} 2024-09-16T04:08:05.137 INFO:tasks.rados.rados.0.smithi016.stdout:27: writing smithi01639990-27 from 0 to 442368 tid 1 2024-09-16T04:08:05.138 INFO:tasks.rados.rados.0.smithi016.stdout:28: write initial oid 28 2024-09-16T04:08:05.138 INFO:tasks.rados.rados.0.smithi016.stdout:28: seq_num 27 ranges {0=458752} 2024-09-16T04:08:05.139 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:05.139 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:05.141 INFO:tasks.rados.rados.0.smithi016.stdout:28: writing smithi01639990-28 from 0 to 458752 tid 1 2024-09-16T04:08:05.141 INFO:tasks.rados.rados.0.smithi016.stdout:29: write initial oid 29 2024-09-16T04:08:05.141 INFO:tasks.rados.rados.0.smithi016.stdout:29: seq_num 28 ranges {0=475136} 2024-09-16T04:08:05.144 INFO:tasks.rados.rados.0.smithi016.stdout:29: writing smithi01639990-29 from 0 to 475136 tid 1 2024-09-16T04:08:05.144 INFO:tasks.rados.rados.0.smithi016.stdout:30: write initial oid 30 2024-09-16T04:08:05.144 INFO:tasks.rados.rados.0.smithi016.stdout:30: seq_num 29 ranges {0=491520} 2024-09-16T04:08:05.148 INFO:tasks.rados.rados.0.smithi016.stdout:30: writing smithi01639990-30 from 0 to 491520 tid 1 2024-09-16T04:08:05.148 INFO:tasks.rados.rados.0.smithi016.stdout:31: write initial oid 31 2024-09-16T04:08:05.148 INFO:tasks.rados.rados.0.smithi016.stdout:31: seq_num 30 ranges {0=507904} 2024-09-16T04:08:05.151 INFO:tasks.rados.rados.0.smithi016.stdout:31: writing smithi01639990-31 from 0 to 507904 tid 1 2024-09-16T04:08:05.151 INFO:tasks.rados.rados.0.smithi016.stdout:32: write initial oid 32 2024-09-16T04:08:05.152 INFO:tasks.rados.rados.0.smithi016.stdout:32: seq_num 31 ranges {0=524288} 2024-09-16T04:08:05.155 INFO:tasks.rados.rados.0.smithi016.stdout:32: writing smithi01639990-32 from 0 to 524288 tid 1 2024-09-16T04:08:05.156 INFO:tasks.rados.rados.0.smithi016.stdout:33: write initial oid 33 2024-09-16T04:08:05.156 INFO:tasks.rados.rados.0.smithi016.stdout:33: seq_num 32 ranges {0=540672} 2024-09-16T04:08:05.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:05.159 INFO:tasks.rados.rados.0.smithi016.stdout:33: writing smithi01639990-33 from 0 to 540672 tid 1 2024-09-16T04:08:05.160 INFO:tasks.rados.rados.0.smithi016.stdout:34: write initial oid 34 2024-09-16T04:08:05.160 INFO:tasks.rados.rados.0.smithi016.stdout:34: seq_num 33 ranges {0=557056} 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:34: writing smithi01639990-34 from 0 to 557056 tid 1 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 16 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:18: finishing write tid 2 to smithi01639990-18 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:22: finishing write tid 1 to smithi01639990-22 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:22: finishing write tid 2 to smithi01639990-22 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:23: finishing write tid 1 to smithi01639990-23 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:23: finishing write tid 2 to smithi01639990-23 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:21: finishing write tid 1 to smithi01639990-21 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:21: finishing write tid 2 to smithi01639990-21 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:23: finishing write tid 3 to smithi01639990-23 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:23: oid 23 updating version 0 to 6 2024-09-16T04:08:05.165 INFO:tasks.rados.rados.0.smithi016.stdout:23: oid 23 version 6 is already newer than 5 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:22: finishing write tid 3 to smithi01639990-22 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:22: oid 22 updating version 0 to 1 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:22: oid 22 updating version 1 to 2 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 22 v 2 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:18: finishing write tid 3 to smithi01639990-18 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:18: oid 18 updating version 0 to 5 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:18: oid 18 updating version 5 to 6 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T04:08:05.166 INFO:tasks.rados.rados.0.smithi016.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:20: finishing write tid 3 to smithi01639990-20 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:20: oid 20 updating version 0 to 6 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:20: oid 20 version 6 is already newer than 5 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:25: finishing write tid 1 to smithi01639990-25 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:18: done (15 left) 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:20: done (14 left) 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:22: done (13 left) 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:23: done (12 left) 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:35: write initial oid 35 2024-09-16T04:08:05.167 INFO:tasks.rados.rados.0.smithi016.stdout:35: seq_num 34 ranges {0=573440} 2024-09-16T04:08:05.168 INFO:tasks.rados.rados.0.smithi016.stdout:35: writing smithi01639990-35 from 0 to 573440 tid 1 2024-09-16T04:08:05.168 INFO:tasks.rados.rados.0.smithi016.stdout:36: write initial oid 36 2024-09-16T04:08:05.168 INFO:tasks.rados.rados.0.smithi016.stdout:36: seq_num 35 ranges {0=589824} 2024-09-16T04:08:05.171 INFO:tasks.rados.rados.0.smithi016.stdout:36: writing smithi01639990-36 from 0 to 589824 tid 1 2024-09-16T04:08:05.171 INFO:tasks.rados.rados.0.smithi016.stdout:37: write initial oid 37 2024-09-16T04:08:05.171 INFO:tasks.rados.rados.0.smithi016.stdout:37: seq_num 36 ranges {0=606208} 2024-09-16T04:08:05.175 INFO:tasks.rados.rados.0.smithi016.stdout:37: writing smithi01639990-37 from 0 to 606208 tid 1 2024-09-16T04:08:05.175 INFO:tasks.rados.rados.0.smithi016.stdout:38: write initial oid 38 2024-09-16T04:08:05.176 INFO:tasks.rados.rados.0.smithi016.stdout:38: seq_num 37 ranges {0=622592} 2024-09-16T04:08:05.177 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:05.175+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:05.182 INFO:tasks.rados.rados.0.smithi016.stdout:38: writing smithi01639990-38 from 0 to 622592 tid 1 2024-09-16T04:08:05.182 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 16 2024-09-16T04:08:05.182 INFO:tasks.rados.rados.0.smithi016.stdout:26: finishing write tid 1 to smithi01639990-26 2024-09-16T04:08:05.182 INFO:tasks.rados.rados.0.smithi016.stdout:21: finishing write tid 3 to smithi01639990-21 2024-09-16T04:08:05.182 INFO:tasks.rados.rados.0.smithi016.stdout:21: oid 21 updating version 0 to 4 2024-09-16T04:08:05.182 INFO:tasks.rados.rados.0.smithi016.stdout:21: oid 21 version 4 is already newer than 3 2024-09-16T04:08:05.182 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 21 v 4 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T04:08:05.182 INFO:tasks.rados.rados.0.smithi016.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-16T04:08:05.182 INFO:tasks.rados.rados.0.smithi016.stdout:26: finishing write tid 2 to smithi01639990-26 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:25: finishing write tid 2 to smithi01639990-25 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:26: finishing write tid 3 to smithi01639990-26 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:26: oid 26 updating version 0 to 1 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:26: oid 26 updating version 1 to 2 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 26 v 2 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:27: finishing write tid 1 to smithi01639990-27 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:27: finishing write tid 2 to smithi01639990-27 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:30: finishing write tid 1 to smithi01639990-30 2024-09-16T04:08:05.183 INFO:tasks.rados.rados.0.smithi016.stdout:27: finishing write tid 3 to smithi01639990-27 2024-09-16T04:08:05.184 INFO:tasks.rados.rados.0.smithi016.stdout:27: oid 27 updating version 0 to 3 2024-09-16T04:08:05.184 INFO:tasks.rados.rados.0.smithi016.stdout:27: oid 27 updating version 3 to 4 2024-09-16T04:08:05.184 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 27 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T04:08:05.184 INFO:tasks.rados.rados.0.smithi016.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-16T04:08:05.184 INFO:tasks.rados.rados.0.smithi016.stdout:30: finishing write tid 2 to smithi01639990-30 2024-09-16T04:08:05.184 INFO:tasks.rados.rados.0.smithi016.stdout:30: finishing write tid 3 to smithi01639990-30 2024-09-16T04:08:05.184 INFO:tasks.rados.rados.0.smithi016.stdout:30: oid 30 updating version 0 to 2 2024-09-16T04:08:05.184 INFO:tasks.rados.rados.0.smithi016.stdout:30: oid 30 version 2 is already newer than 1 2024-09-16T04:08:05.184 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 30 v 2 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:24: finishing write tid 1 to smithi01639990-24 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:28: finishing write tid 1 to smithi01639990-28 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:24: finishing write tid 2 to smithi01639990-24 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:24: finishing write tid 3 to smithi01639990-24 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:24: oid 24 updating version 0 to 4 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:24: oid 24 version 4 is already newer than 3 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-16T04:08:05.185 INFO:tasks.rados.rados.0.smithi016.stdout:29: finishing write tid 1 to smithi01639990-29 2024-09-16T04:08:05.186 INFO:tasks.rados.rados.0.smithi016.stdout:25: finishing write tid 3 to smithi01639990-25 2024-09-16T04:08:05.186 INFO:tasks.rados.rados.0.smithi016.stdout:25: oid 25 updating version 0 to 4 2024-09-16T04:08:05.186 INFO:tasks.rados.rados.0.smithi016.stdout:25: oid 25 version 4 is already newer than 3 2024-09-16T04:08:05.186 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T04:08:05.186 INFO:tasks.rados.rados.0.smithi016.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-16T04:08:05.186 INFO:tasks.rados.rados.0.smithi016.stdout:32: finishing write tid 1 to smithi01639990-32 2024-09-16T04:08:05.186 INFO:tasks.rados.rados.0.smithi016.stdout:29: finishing write tid 2 to smithi01639990-29 2024-09-16T04:08:05.186 INFO:tasks.rados.rados.0.smithi016.stdout:33: finishing write tid 1 to smithi01639990-33 2024-09-16T04:08:05.186 INFO:tasks.rados.rados.0.smithi016.stdout:32: finishing write tid 2 to smithi01639990-32 2024-09-16T04:08:05.187 INFO:tasks.rados.rados.0.smithi016.stdout:33: finishing write tid 2 to smithi01639990-33 2024-09-16T04:08:05.187 INFO:tasks.rados.rados.0.smithi016.stdout:29: finishing write tid 3 to smithi01639990-29 2024-09-16T04:08:05.187 INFO:tasks.rados.rados.0.smithi016.stdout:29: oid 29 updating version 0 to 3 2024-09-16T04:08:05.187 INFO:tasks.rados.rados.0.smithi016.stdout:29: oid 29 updating version 3 to 4 2024-09-16T04:08:05.187 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 29 v 4 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T04:08:05.187 INFO:tasks.rados.rados.0.smithi016.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-16T04:08:05.187 INFO:tasks.rados.rados.0.smithi016.stdout:34: finishing write tid 1 to smithi01639990-34 2024-09-16T04:08:05.187 INFO:tasks.rados.rados.0.smithi016.stdout:34: finishing write tid 2 to smithi01639990-34 2024-09-16T04:08:05.187 INFO:tasks.rados.rados.0.smithi016.stdout:35: finishing write tid 1 to smithi01639990-35 2024-09-16T04:08:05.188 INFO:tasks.rados.rados.0.smithi016.stdout:35: finishing write tid 2 to smithi01639990-35 2024-09-16T04:08:05.188 INFO:tasks.rados.rados.0.smithi016.stdout:33: finishing write tid 3 to smithi01639990-33 2024-09-16T04:08:05.188 INFO:tasks.rados.rados.0.smithi016.stdout:33: oid 33 updating version 0 to 7 2024-09-16T04:08:05.188 INFO:tasks.rados.rados.0.smithi016.stdout:33: oid 33 updating version 7 to 8 2024-09-16T04:08:05.188 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 33 v 8 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T04:08:05.188 INFO:tasks.rados.rados.0.smithi016.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:08:05.188 INFO:tasks.rados.rados.0.smithi016.stdout:34: finishing write tid 3 to smithi01639990-34 2024-09-16T04:08:05.188 INFO:tasks.rados.rados.0.smithi016.stdout:34: oid 34 updating version 0 to 5 2024-09-16T04:08:05.188 INFO:tasks.rados.rados.0.smithi016.stdout:34: oid 34 updating version 5 to 6 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:28: finishing write tid 2 to smithi01639990-28 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:21: done (15 left) 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:24: done (14 left) 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:25: done (13 left) 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:26: done (12 left) 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:27: done (11 left) 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:29: done (10 left) 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:30: done (9 left) 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:33: done (8 left) 2024-09-16T04:08:05.189 INFO:tasks.rados.rados.0.smithi016.stdout:34: done (7 left) 2024-09-16T04:08:05.190 INFO:tasks.rados.rados.0.smithi016.stdout:39: write initial oid 39 2024-09-16T04:08:05.190 INFO:tasks.rados.rados.0.smithi016.stdout:39: seq_num 38 ranges {0=638976} 2024-09-16T04:08:05.190 INFO:tasks.rados.rados.0.smithi016.stdout:39: writing smithi01639990-39 from 0 to 638976 tid 1 2024-09-16T04:08:05.191 INFO:tasks.rados.rados.0.smithi016.stdout:40: write initial oid 40 2024-09-16T04:08:05.191 INFO:tasks.rados.rados.0.smithi016.stdout:40: seq_num 39 ranges {0=671744} 2024-09-16T04:08:05.191 INFO:tasks.rados.rados.0.smithi016.stdout:40: writing smithi01639990-40 from 0 to 671744 tid 1 2024-09-16T04:08:05.191 INFO:tasks.rados.rados.0.smithi016.stdout:41: write initial oid 41 2024-09-16T04:08:05.191 INFO:tasks.rados.rados.0.smithi016.stdout:41: seq_num 40 ranges {0=688128} 2024-09-16T04:08:05.194 INFO:tasks.rados.rados.0.smithi016.stdout:41: writing smithi01639990-41 from 0 to 688128 tid 1 2024-09-16T04:08:05.194 INFO:tasks.rados.rados.0.smithi016.stdout:36: finishing write tid 1 to smithi01639990-36 2024-09-16T04:08:05.194 INFO:tasks.rados.rados.0.smithi016.stdout:36: finishing write tid 2 to smithi01639990-36 2024-09-16T04:08:05.194 INFO:tasks.rados.rados.0.smithi016.stdout:28: finishing write tid 3 to smithi01639990-28 2024-09-16T04:08:05.194 INFO:tasks.rados.rados.0.smithi016.stdout:28: oid 28 updating version 0 to 7 2024-09-16T04:08:05.194 INFO:tasks.rados.rados.0.smithi016.stdout:28: oid 28 updating version 7 to 8 2024-09-16T04:08:05.194 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 28 v 8 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T04:08:05.194 INFO:tasks.rados.rados.0.smithi016.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-16T04:08:05.194 INFO:tasks.rados.rados.0.smithi016.stdout:37: finishing write tid 1 to smithi01639990-37 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:37: finishing write tid 2 to smithi01639990-37 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:38: finishing write tid 1 to smithi01639990-38 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:38: finishing write tid 2 to smithi01639990-38 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:37: finishing write tid 3 to smithi01639990-37 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:37: oid 37 updating version 0 to 6 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:37: oid 37 version 6 is already newer than 5 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:38: finishing write tid 3 to smithi01639990-38 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:38: oid 38 updating version 0 to 3 2024-09-16T04:08:05.195 INFO:tasks.rados.rados.0.smithi016.stdout:38: oid 38 updating version 3 to 4 2024-09-16T04:08:05.196 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 38 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T04:08:05.196 INFO:tasks.rados.rados.0.smithi016.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-16T04:08:05.196 INFO:tasks.rados.rados.0.smithi016.stdout:39: finishing write tid 1 to smithi01639990-39 2024-09-16T04:08:05.196 INFO:tasks.rados.rados.0.smithi016.stdout:39: finishing write tid 2 to smithi01639990-39 2024-09-16T04:08:05.196 INFO:tasks.rados.rados.0.smithi016.stdout:28: done (9 left) 2024-09-16T04:08:05.196 INFO:tasks.rados.rados.0.smithi016.stdout:37: done (8 left) 2024-09-16T04:08:05.196 INFO:tasks.rados.rados.0.smithi016.stdout:38: done (7 left) 2024-09-16T04:08:05.196 INFO:tasks.rados.rados.0.smithi016.stdout:42: write initial oid 42 2024-09-16T04:08:05.196 INFO:tasks.rados.rados.0.smithi016.stdout:42: seq_num 41 ranges {0=704512} 2024-09-16T04:08:05.199 INFO:tasks.rados.rados.0.smithi016.stdout:42: writing smithi01639990-42 from 0 to 704512 tid 1 2024-09-16T04:08:05.199 INFO:tasks.rados.rados.0.smithi016.stdout:39: finishing write tid 3 to smithi01639990-39 2024-09-16T04:08:05.199 INFO:tasks.rados.rados.0.smithi016.stdout:39: oid 39 updating version 0 to 4 2024-09-16T04:08:05.199 INFO:tasks.rados.rados.0.smithi016.stdout:39: oid 39 version 4 is already newer than 3 2024-09-16T04:08:05.199 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 39 v 4 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T04:08:05.199 INFO:tasks.rados.rados.0.smithi016.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-16T04:08:05.199 INFO:tasks.rados.rados.0.smithi016.stdout:40: finishing write tid 1 to smithi01639990-40 2024-09-16T04:08:05.199 INFO:tasks.rados.rados.0.smithi016.stdout:40: finishing write tid 2 to smithi01639990-40 2024-09-16T04:08:05.199 INFO:tasks.rados.rados.0.smithi016.stdout:39: done (7 left) 2024-09-16T04:08:05.200 INFO:tasks.rados.rados.0.smithi016.stdout:43: write initial oid 43 2024-09-16T04:08:05.200 INFO:tasks.rados.rados.0.smithi016.stdout:43: seq_num 42 ranges {0=720896} 2024-09-16T04:08:05.203 INFO:tasks.rados.rados.0.smithi016.stdout:43: writing smithi01639990-43 from 0 to 720896 tid 1 2024-09-16T04:08:05.203 INFO:tasks.rados.rados.0.smithi016.stdout:44: write initial oid 44 2024-09-16T04:08:05.203 INFO:tasks.rados.rados.0.smithi016.stdout:44: seq_num 43 ranges {0=737280} 2024-09-16T04:08:05.208 INFO:tasks.rados.rados.0.smithi016.stdout:44: writing smithi01639990-44 from 0 to 737280 tid 1 2024-09-16T04:08:05.208 INFO:tasks.rados.rados.0.smithi016.stdout:45: write initial oid 45 2024-09-16T04:08:05.208 INFO:tasks.rados.rados.0.smithi016.stdout:45: seq_num 44 ranges {0=753664} 2024-09-16T04:08:05.213 INFO:tasks.rados.rados.0.smithi016.stdout:45: writing smithi01639990-45 from 0 to 753664 tid 1 2024-09-16T04:08:05.213 INFO:tasks.rados.rados.0.smithi016.stdout:46: write initial oid 46 2024-09-16T04:08:05.213 INFO:tasks.rados.rados.0.smithi016.stdout:46: seq_num 45 ranges {0=770048} 2024-09-16T04:08:05.219 INFO:tasks.rados.rados.0.smithi016.stdout:46: writing smithi01639990-46 from 0 to 770048 tid 1 2024-09-16T04:08:05.219 INFO:tasks.rados.rados.0.smithi016.stdout:47: write initial oid 47 2024-09-16T04:08:05.219 INFO:tasks.rados.rados.0.smithi016.stdout:47: seq_num 46 ranges {0=786432} 2024-09-16T04:08:05.225 INFO:tasks.rados.rados.0.smithi016.stdout:47: writing smithi01639990-47 from 0 to 786432 tid 1 2024-09-16T04:08:05.225 INFO:tasks.rados.rados.0.smithi016.stdout:48: write initial oid 48 2024-09-16T04:08:05.225 INFO:tasks.rados.rados.0.smithi016.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-09-16T04:08:05.227 INFO:tasks.rados.rados.0.smithi016.stdout:48: writing smithi01639990-48 from 0 to 425984 tid 1 2024-09-16T04:08:05.230 INFO:tasks.rados.rados.0.smithi016.stdout:48: writing smithi01639990-48 from 425984 to 802816 tid 2 2024-09-16T04:08:05.230 INFO:tasks.rados.rados.0.smithi016.stdout:49: write initial oid 49 2024-09-16T04:08:05.230 INFO:tasks.rados.rados.0.smithi016.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-09-16T04:08:05.233 INFO:tasks.rados.rados.0.smithi016.stdout:49: writing smithi01639990-49 from 0 to 442368 tid 1 2024-09-16T04:08:05.236 INFO:tasks.rados.rados.0.smithi016.stdout:49: writing smithi01639990-49 from 442368 to 819200 tid 2 2024-09-16T04:08:05.236 INFO:tasks.rados.rados.0.smithi016.stdout:50: write initial oid 50 2024-09-16T04:08:05.237 INFO:tasks.rados.rados.0.smithi016.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-09-16T04:08:05.240 INFO:tasks.rados.rados.0.smithi016.stdout:50: writing smithi01639990-50 from 0 to 458752 tid 1 2024-09-16T04:08:05.244 INFO:tasks.rados.rados.0.smithi016.stdout:50: writing smithi01639990-50 from 458752 to 835584 tid 2 2024-09-16T04:08:05.244 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 16 current snap is 0 2024-09-16T04:08:05.244 INFO:tasks.rados.rados.0.smithi016.stdout:41: finishing write tid 1 to smithi01639990-41 2024-09-16T04:08:05.244 INFO:tasks.rados.rados.0.smithi016.stdout:40: finishing write tid 3 to smithi01639990-40 2024-09-16T04:08:05.244 INFO:tasks.rados.rados.0.smithi016.stdout:40: oid 40 updating version 0 to 6 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:40: oid 40 version 6 is already newer than 5 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:32: finishing write tid 3 to smithi01639990-32 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:32: oid 32 updating version 0 to 5 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:32: oid 32 updating version 5 to 6 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:42: finishing write tid 1 to smithi01639990-42 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:42: finishing write tid 2 to smithi01639990-42 2024-09-16T04:08:05.245 INFO:tasks.rados.rados.0.smithi016.stdout:41: finishing write tid 2 to smithi01639990-41 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:36: finishing write tid 3 to smithi01639990-36 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:36: oid 36 updating version 0 to 6 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:36: oid 36 version 6 is already newer than 5 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:43: finishing write tid 1 to smithi01639990-43 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:44: finishing write tid 1 to smithi01639990-44 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:41: finishing write tid 3 to smithi01639990-41 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:41: oid 41 updating version 0 to 7 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:41: oid 41 updating version 7 to 8 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 41 v 8 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-16T04:08:05.246 INFO:tasks.rados.rados.0.smithi016.stdout:45: finishing write tid 1 to smithi01639990-45 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:45: finishing write tid 2 to smithi01639990-45 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:45: finishing write tid 3 to smithi01639990-45 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:45: oid 45 updating version 0 to 8 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:45: oid 45 version 8 is already newer than 7 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 45 v 8 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:43: finishing write tid 2 to smithi01639990-43 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:43: finishing write tid 3 to smithi01639990-43 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:43: oid 43 updating version 0 to 10 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:43: oid 43 version 10 is already newer than 9 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 43 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:46: finishing write tid 1 to smithi01639990-46 2024-09-16T04:08:05.247 INFO:tasks.rados.rados.0.smithi016.stdout:51: seq_num 50 ranges {262144=475136,737280=376832} 2024-09-16T04:08:05.248 INFO:tasks.rados.rados.0.smithi016.stdout:51: writing smithi01639990-16 from 262144 to 737280 tid 1 2024-09-16T04:08:05.250 INFO:tasks.rados.rados.0.smithi016.stdout:51: writing smithi01639990-16 from 737280 to 1114112 tid 2 2024-09-16T04:08:05.250 INFO:tasks.rados.rados.0.smithi016.stdout:32: done (15 left) 2024-09-16T04:08:05.250 INFO:tasks.rados.rados.0.smithi016.stdout:36: done (14 left) 2024-09-16T04:08:05.250 INFO:tasks.rados.rados.0.smithi016.stdout:40: done (13 left) 2024-09-16T04:08:05.250 INFO:tasks.rados.rados.0.smithi016.stdout:41: done (12 left) 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:43: done (11 left) 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:45: done (10 left) 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 34 current snap is 0 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:47: finishing write tid 1 to smithi01639990-47 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:47: finishing write tid 2 to smithi01639990-47 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:44: finishing write tid 2 to smithi01639990-44 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:46: finishing write tid 2 to smithi01639990-46 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:47: finishing write tid 3 to smithi01639990-47 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:47: oid 47 updating version 0 to 6 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:47: oid 47 version 6 is already newer than 5 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:44: finishing write tid 3 to smithi01639990-44 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:44: oid 44 updating version 0 to 7 2024-09-16T04:08:05.251 INFO:tasks.rados.rados.0.smithi016.stdout:44: oid 44 updating version 7 to 8 2024-09-16T04:08:05.252 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 44 v 8 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T04:08:05.252 INFO:tasks.rados.rados.0.smithi016.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-16T04:08:05.252 INFO:tasks.rados.rados.0.smithi016.stdout:52: seq_num 51 ranges {557056=491520,1048576=376832} 2024-09-16T04:08:05.256 INFO:tasks.rados.rados.0.smithi016.stdout:52: writing smithi01639990-34 from 557056 to 1048576 tid 1 2024-09-16T04:08:05.260 INFO:tasks.rados.rados.0.smithi016.stdout:52: writing smithi01639990-34 from 1048576 to 1425408 tid 2 2024-09-16T04:08:05.260 INFO:tasks.rados.rados.0.smithi016.stdout:44: done (10 left) 2024-09-16T04:08:05.260 INFO:tasks.rados.rados.0.smithi016.stdout:47: done (9 left) 2024-09-16T04:08:05.261 INFO:tasks.rados.rados.0.smithi016.stdout:53: copy_from oid 18 from oid 37 current snap is 0 2024-09-16T04:08:05.261 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 28 current snap is 0 2024-09-16T04:08:05.261 INFO:tasks.rados.rados.0.smithi016.stdout:54: seq_num 52 ranges {458752=507904,966656=376832} 2024-09-16T04:08:05.264 INFO:tasks.rados.rados.0.smithi016.stdout:54: writing smithi01639990-28 from 458752 to 966656 tid 1 2024-09-16T04:08:05.266 INFO:tasks.rados.rados.0.smithi016.stdout:54: writing smithi01639990-28 from 966656 to 1343488 tid 2 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:55: snap_create 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:46: finishing write tid 3 to smithi01639990-46 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:46: oid 46 updating version 0 to 11 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:46: oid 46 updating version 11 to 12 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:48: finishing write tid 1 to smithi01639990-48 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:48: finishing write tid 2 to smithi01639990-48 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:49: finishing write tid 1 to smithi01639990-49 2024-09-16T04:08:05.267 INFO:tasks.rados.rados.0.smithi016.stdout:49: finishing write tid 2 to smithi01639990-49 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:50: finishing write tid 1 to smithi01639990-50 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:50: finishing write tid 2 to smithi01639990-50 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:50: finishing write tid 3 to smithi01639990-50 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:51: finishing write tid 1 to smithi01639990-16 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:50: finishing write tid 4 to smithi01639990-50 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:50: oid 50 updating version 0 to 9 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:50: oid 50 version 9 is already newer than 8 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:50: oid 50 version 9 is already newer than 7 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:51: finishing write tid 2 to smithi01639990-16 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:51: finishing write tid 3 to smithi01639990-16 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:51: finishing write tid 4 to smithi01639990-16 2024-09-16T04:08:05.268 INFO:tasks.rados.rados.0.smithi016.stdout:51: oid 16 updating version 0 to 10 2024-09-16T04:08:05.269 INFO:tasks.rados.rados.0.smithi016.stdout:51: oid 16 updating version 10 to 11 2024-09-16T04:08:05.269 INFO:tasks.rados.rados.0.smithi016.stdout:51: oid 16 updating version 11 to 12 2024-09-16T04:08:05.269 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 16 v 12 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-16T04:08:05.269 INFO:tasks.rados.rados.0.smithi016.stdout:51: left oid 16 (ObjNum 50 snap 0 seq_num 50) 2024-09-16T04:08:05.269 INFO:tasks.rados.rados.0.smithi016.stdout:52: finishing write tid 1 to smithi01639990-34 2024-09-16T04:08:05.271 INFO:tasks.rados.rados.0.smithi016.stdout:53: finishing copy_from racing read to smithi01639990-18 2024-09-16T04:08:05.272 INFO:tasks.rados.rados.0.smithi016.stdout:49: finishing write tid 3 to smithi01639990-49 2024-09-16T04:08:05.273 INFO:tasks.rados.rados.0.smithi016.stdout:48: finishing write tid 3 to smithi01639990-48 2024-09-16T04:08:05.273 INFO:tasks.rados.rados.0.smithi016.stdout:52: finishing write tid 2 to smithi01639990-34 2024-09-16T04:08:05.273 INFO:tasks.rados.rados.0.smithi016.stdout:52: finishing write tid 3 to smithi01639990-34 2024-09-16T04:08:05.274 INFO:tasks.rados.rados.0.smithi016.stdout:54: finishing write tid 1 to smithi01639990-28 2024-09-16T04:08:05.276 INFO:tasks.rados.rados.0.smithi016.stdout:54: finishing write tid 2 to smithi01639990-28 2024-09-16T04:08:05.276 INFO:tasks.rados.rados.0.smithi016.stdout:53: finishing copy_from to smithi01639990-18 2024-09-16T04:08:05.276 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 18 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:08:05.278 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:05.276+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:05.278 INFO:tasks.rados.rados.0.smithi016.stdout:54: finishing write tid 3 to smithi01639990-28 2024-09-16T04:08:05.279 INFO:tasks.rados.rados.0.smithi016.stdout:49: finishing write tid 4 to smithi01639990-49 2024-09-16T04:08:05.279 INFO:tasks.rados.rados.0.smithi016.stdout:49: oid 49 updating version 0 to 11 2024-09-16T04:08:05.279 INFO:tasks.rados.rados.0.smithi016.stdout:49: oid 49 version 11 is already newer than 10 2024-09-16T04:08:05.279 INFO:tasks.rados.rados.0.smithi016.stdout:49: oid 49 version 11 is already newer than 9 2024-09-16T04:08:05.280 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 49 v 11 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T04:08:05.280 INFO:tasks.rados.rados.0.smithi016.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-16T04:08:05.280 INFO:tasks.rados.rados.0.smithi016.stdout:48: finishing write tid 4 to smithi01639990-48 2024-09-16T04:08:05.280 INFO:tasks.rados.rados.0.smithi016.stdout:48: oid 48 updating version 0 to 9 2024-09-16T04:08:05.280 INFO:tasks.rados.rados.0.smithi016.stdout:48: oid 48 updating version 9 to 11 2024-09-16T04:08:05.280 INFO:tasks.rados.rados.0.smithi016.stdout:48: oid 48 version 11 is already newer than 10 2024-09-16T04:08:05.280 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 48 v 11 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T04:08:05.280 INFO:tasks.rados.rados.0.smithi016.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-16T04:08:05.305 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:05.306 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:05.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:05.392 INFO:tasks.rados.rados.0.smithi016.stdout:52: finishing write tid 4 to smithi01639990-34 2024-09-16T04:08:05.392 INFO:tasks.rados.rados.0.smithi016.stdout:52: oid 34 updating version 0 to 13 2024-09-16T04:08:05.392 INFO:tasks.rados.rados.0.smithi016.stdout:52: oid 34 updating version 13 to 14 2024-09-16T04:08:05.392 INFO:tasks.rados.rados.0.smithi016.stdout:52: oid 34 updating version 14 to 15 2024-09-16T04:08:05.392 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 34 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-16T04:08:05.392 INFO:tasks.rados.rados.0.smithi016.stdout:52: left oid 34 (ObjNum 51 snap 0 seq_num 51) 2024-09-16T04:08:05.393 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:05.390+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:05.461 INFO:tasks.rados.rados.0.smithi016.stdout:42: finishing write tid 3 to smithi01639990-42 2024-09-16T04:08:05.461 INFO:tasks.rados.rados.0.smithi016.stdout:42: oid 42 updating version 0 to 10 2024-09-16T04:08:05.461 INFO:tasks.rados.rados.0.smithi016.stdout:42: oid 42 version 10 is already newer than 9 2024-09-16T04:08:05.461 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 42 v 10 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T04:08:05.461 INFO:tasks.rados.rados.0.smithi016.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-16T04:08:05.464 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:05.464 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:05.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:05.514 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:05.512+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:05.580 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:05.579+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:05.621 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:05.621 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:05.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:05.682 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:05.681+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:05.736 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:05.736 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:05.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:05.785 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:05.782+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:05.811 INFO:tasks.rados.rados.0.smithi016.stdout:35: finishing write tid 3 to smithi01639990-35 2024-09-16T04:08:05.811 INFO:tasks.rados.rados.0.smithi016.stdout:35: oid 35 updating version 0 to 3 2024-09-16T04:08:05.811 INFO:tasks.rados.rados.0.smithi016.stdout:35: oid 35 updating version 3 to 4 2024-09-16T04:08:05.811 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 35 v 4 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T04:08:05.812 INFO:tasks.rados.rados.0.smithi016.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-16T04:08:05.846 INFO:tasks.rados.rados.0.smithi016.stdout:31: finishing write tid 1 to smithi01639990-31 2024-09-16T04:08:05.852 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:05.852 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:05.853 INFO:tasks.rados.rados.0.smithi016.stdout:54: finishing write tid 4 to smithi01639990-28 2024-09-16T04:08:05.853 INFO:tasks.rados.rados.0.smithi016.stdout:54: oid 28 updating version 0 to 15 2024-09-16T04:08:05.853 INFO:tasks.rados.rados.0.smithi016.stdout:54: oid 28 version 15 is already newer than 13 2024-09-16T04:08:05.853 INFO:tasks.rados.rados.0.smithi016.stdout:54: oid 28 version 15 is already newer than 12 2024-09-16T04:08:05.853 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 28 v 15 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-16T04:08:05.853 INFO:tasks.rados.rados.0.smithi016.stdout:54: left oid 28 (ObjNum 52 snap 0 seq_num 52) 2024-09-16T04:08:05.861 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:05.885 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:05.883+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:05.909 INFO:tasks.rados.rados.0.smithi016.stdout:31: finishing write tid 2 to smithi01639990-31 2024-09-16T04:08:05.966 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:05.966 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:05.985 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:05.983+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:set 3.d pg_upmap mapping to [7,8,4,11,9,0,1,5] 2024-09-16T04:08:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:35: done (11 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:42: done (10 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:46: done (9 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:48: done (8 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:49: done (7 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:50: done (6 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:51: done (5 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:52: done (4 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:53: done (3 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:54: done (2 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:55: done (1 left) 2024-09-16T04:08:06.081 INFO:tasks.rados.rados.0.smithi016.stdout:56: rmattr oid 11 current snap is 1 2024-09-16T04:08:06.082 INFO:tasks.rados.rados.0.smithi016.stdout:56: done (1 left) 2024-09-16T04:08:06.082 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 34 current snap is 1 2024-09-16T04:08:06.082 INFO:tasks.rados.rados.0.smithi016.stdout:57: seq_num 53 ranges {1425408=524288,1949696=376832} 2024-09-16T04:08:06.087 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:06.084+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:06.091 INFO:tasks.rados.rados.0.smithi016.stdout:57: writing smithi01639990-34 from 1425408 to 1949696 tid 1 2024-09-16T04:08:06.094 INFO:tasks.rados.rados.0.smithi016.stdout:57: writing smithi01639990-34 from 1949696 to 2326528 tid 2 2024-09-16T04:08:06.094 INFO:tasks.rados.rados.0.smithi016.stdout:58: read oid 28 snap -1 2024-09-16T04:08:06.095 INFO:tasks.rados.rados.0.smithi016.stdout:58: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T04:08:06.095 INFO:tasks.rados.rados.0.smithi016.stdout:59: rollback oid 47 current snap is 1 2024-09-16T04:08:06.095 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 47 to 0 2024-09-16T04:08:06.095 INFO:tasks.rados.rados.0.smithi016.stdout:60: rmattr oid 44 current snap is 1 2024-09-16T04:08:06.095 INFO:tasks.rados.rados.0.smithi016.stdout:60: done (4 left) 2024-09-16T04:08:06.095 INFO:tasks.rados.rados.0.smithi016.stdout:61: snap_create 2024-09-16T04:08:06.101 INFO:tasks.rados.rados.0.smithi016.stdout:59: finishing rollback tid 0 to smithi01639990-47 2024-09-16T04:08:06.101 INFO:tasks.rados.rados.0.smithi016.stdout:57: finishing write tid 1 to smithi01639990-34 2024-09-16T04:08:06.101 INFO:tasks.rados.rados.0.smithi016.stdout:57: finishing write tid 2 to smithi01639990-34 2024-09-16T04:08:06.101 INFO:tasks.rados.rados.0.smithi016.stdout:59: finishing rollback tid 1 to smithi01639990-47 2024-09-16T04:08:06.101 INFO:tasks.rados.rados.0.smithi016.stdout:59: finishing rollback tid 2 to smithi01639990-47 2024-09-16T04:08:06.102 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 47 v 10 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T04:08:06.103 INFO:tasks.rados.rados.0.smithi016.stdout:58: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T04:08:06.146 INFO:tasks.rados.rados.0.smithi016.stdout:31: finishing write tid 3 to smithi01639990-31 2024-09-16T04:08:06.146 INFO:tasks.rados.rados.0.smithi016.stdout:31: oid 31 updating version 0 to 6 2024-09-16T04:08:06.147 INFO:tasks.rados.rados.0.smithi016.stdout:31: oid 31 version 6 is already newer than 5 2024-09-16T04:08:06.147 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T04:08:06.147 INFO:tasks.rados.rados.0.smithi016.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-16T04:08:06.184 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:06.187 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:06.185+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:06.286 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:06.287 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:06.288 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:06.286+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:06.295 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:06.389 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:06.387+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:06.398 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:06.398 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:06.407 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:06.491 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:06.489+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:06.509 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:06.509 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:06.518 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:06.592 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:06.590+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:06.620 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:06.620 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:06.629 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:06.692 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:06.691+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:06.731 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:06.732 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:06.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:06.740 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:06.793 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:06.791+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:06.796 INFO:tasks.rados.rados.0.smithi016.stdout:57: finishing write tid 3 to smithi01639990-34 2024-09-16T04:08:06.844 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:06.844 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:06.845 INFO:tasks.rados.rados.0.smithi016.stdout:57: finishing write tid 4 to smithi01639990-34 2024-09-16T04:08:06.845 INFO:tasks.rados.rados.0.smithi016.stdout:57: oid 34 updating version 0 to 17 2024-09-16T04:08:06.845 INFO:tasks.rados.rados.0.smithi016.stdout:57: oid 34 updating version 17 to 19 2024-09-16T04:08:06.846 INFO:tasks.rados.rados.0.smithi016.stdout:57: oid 34 version 19 is already newer than 18 2024-09-16T04:08:06.846 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 34 v 19 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-09-16T04:08:06.846 INFO:tasks.rados.rados.0.smithi016.stdout:57: left oid 34 (ObjNum 53 snap 1 seq_num 53) 2024-09-16T04:08:06.852 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:06.879 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:06.894 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:06.892+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:06.900 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:06.900 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:06.921 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:06.921 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:06.942 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:06.955 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:06.956 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:06.963 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:06.963 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:06.964 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:06.984 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:06.995 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:06.993+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:07.067 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:07.067 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:07.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:07.067 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [] 2024-09-16T04:08:07.067 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:07.075 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:07.084 INFO:tasks.rados.rados.0.smithi016.stdout:31: done (4 left) 2024-09-16T04:08:07.085 INFO:tasks.rados.rados.0.smithi016.stdout:57: done (3 left) 2024-09-16T04:08:07.085 INFO:tasks.rados.rados.0.smithi016.stdout:58: done (2 left) 2024-09-16T04:08:07.085 INFO:tasks.rados.rados.0.smithi016.stdout:59: done (1 left) 2024-09-16T04:08:07.085 INFO:tasks.rados.rados.0.smithi016.stdout:61: done (0 left) 2024-09-16T04:08:07.085 INFO:tasks.rados.rados.0.smithi016.stdout:62: delete oid 18 current snap is 2 2024-09-16T04:08:07.086 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.085+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.087 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.085+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.087 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.085+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.087 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.085+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.088 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.086+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.088 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.086+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.088 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.086+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.088 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.086+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.088 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.086+0000 7f650f91d640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.088 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.086+0000 7f650f91d640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.088 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.086+0000 7f650f91d640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.088 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.086+0000 7f650f91d640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.089 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.087+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.089 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.087+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.090 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.088+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.090 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.088+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.090 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.088+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.090 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.088+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.090 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.089+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.090 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.089+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.092 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.090+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.092 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.090+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:07.093 INFO:tasks.rados.rados.0.smithi016.stdout:62: done (0 left) 2024-09-16T04:08:07.094 INFO:tasks.rados.rados.0.smithi016.stdout:63: delete oid 19 current snap is 2 2024-09-16T04:08:07.096 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:07.094+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:07.102 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:07.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:07.148 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:07.170 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:07.170 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:07.184 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:07.185 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:07.193 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:07.194 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:07.194 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:07.196 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:07.194+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:07.216 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:07.297 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:07.296+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:07.337 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:07.337 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:07.337 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:07.337 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:07.337 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:07.345 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:07.398 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:07.397+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:07.450 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.506903297000001, 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.451 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:07.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:07.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.457 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923042+0000", 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.524584592, 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075287749999999997, 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 35 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923042+0000", 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923042+0000", 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923042+0000", 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2700000000000001e-07 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923097+0000", 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4897e-05 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923103+0000", 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0299999999999999e-06 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923106+0000", 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.591e-06 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.928324+0000", 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005217742 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.928428+0000", 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010402499999999999 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.928560+0000", 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000131698 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930427+0000", 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001867226 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930470+0000", 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3241999999999997e-05 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930529+0000", 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8576999999999998e-05 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930570+0000", 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1520000000000002e-05 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.463 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923113+0000", 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.524513637999998, 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0076822339999999996, 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.464 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 36 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923113+0000", 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923113+0000", 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923113+0000", 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4999999999999998e-07 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923115+0000", 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4420000000000001e-06 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923120+0000", 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.211e-06 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923123+0000", 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.332e-06 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.929534+0000", 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0064114539999999996 2024-09-16T04:08:07.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.929635+0000", 2024-09-16T04:08:07.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000100299 2024-09-16T04:08:07.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.929770+0000", 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000135351 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930683+0000", 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00091337999999999999 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930724+0000", 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1063999999999999e-05 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930757+0000", 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2376999999999998e-05 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930795+0000", 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7973999999999999e-05 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:07.470 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.524499054, 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:07.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.514136548, 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.476 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:57 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:51.800186+0000", 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.647440016999999, 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075740590000000002, 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.480 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 57 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800186+0000", 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800186+0000", 2024-09-16T04:08:07.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800187+0000", 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1650000000000001e-06 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800225+0000", 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7923999999999998e-05 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800236+0000", 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0658e-05 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800243+0000", 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0389999999999999e-06 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800281+0000", 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7764e-05 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800323+0000", 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2256000000000003e-05 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800395+0000", 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1962e-05 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.802001+0000", 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016061719999999999 2024-09-16T04:08:07.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.807705+0000", 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057043279999999998 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.807737+0000", 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1322999999999998e-05 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.807760+0000", 2024-09-16T04:08:07.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3468e-05 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:58 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:51.800265+0000", 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.647361438000001, 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086104170000000004, 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.486 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 58 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800265+0000", 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800265+0000", 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800265+0000", 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0100000000000004e-07 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800353+0000", 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7968999999999998e-05 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800363+0000", 2024-09-16T04:08:07.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.815e-06 2024-09-16T04:08:07.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800371+0000", 2024-09-16T04:08:07.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5220000000000001e-06 2024-09-16T04:08:07.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800725+0000", 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035470800000000002 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800776+0000", 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0625999999999999e-05 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800859+0000", 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2812999999999995e-05 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.802211+0000", 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013517519999999999 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.808815+0000", 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0066041909999999997 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.808850+0000", 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4761999999999997e-05 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.808875+0000", 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5558000000000002e-05 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.492 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:51.800386+0000", 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.647239770000001, 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086667150000000002, 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.493 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 59 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800386+0000", 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800386+0000", 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800387+0000", 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9999999999999997e-07 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800392+0000", 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.899e-06 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800400+0000", 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5410000000000003e-06 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800405+0000", 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.823e-06 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.801125+0000", 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00071925799999999996 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.801175+0000", 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0037999999999998e-05 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.801263+0000", 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8392000000000003e-05 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.802218+0000", 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095508799999999999 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.809001+0000", 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067828769999999997 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.809031+0000", 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9920000000000002e-05 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.809053+0000", 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2178999999999999e-05 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:51.825046+0000", 2024-09-16T04:08:07.499 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.622580112, 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082050490000000007, 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 88 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825046+0000", 2024-09-16T04:08:07.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825046+0000", 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825046+0000", 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5699999999999998e-07 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825049+0000", 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0130000000000002e-06 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825054+0000", 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6090000000000003e-06 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.825057+0000", 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2109999999999998e-06 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.827954+0000", 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002897125 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.828005+0000", 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0847999999999998e-05 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.828085+0000", 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9419000000000007e-05 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.828558+0000", 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00047363100000000002 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.833195+0000", 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004636444 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.833227+0000", 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.201e-05 2024-09-16T04:08:07.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.833251+0000", 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4382000000000001e-05 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.017548496, 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.505 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:07.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:07.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:07.510 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.453512+0000", 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.994113778999999, 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069585710000000002, 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.511 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 140 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453512+0000", 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453512+0000", 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453513+0000", 2024-09-16T04:08:07.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:07.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453565+0000", 2024-09-16T04:08:07.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1987000000000002e-05 2024-09-16T04:08:07.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453573+0000", 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4309999999999994e-06 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453580+0000", 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5069999999999997e-06 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.458788+0000", 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005207761 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.458832+0000", 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3890000000000002e-05 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.458904+0000", 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2004999999999999e-05 2024-09-16T04:08:07.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459640+0000", 2024-09-16T04:08:07.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00073623500000000001 2024-09-16T04:08:07.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460428+0000", 2024-09-16T04:08:07.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00078837600000000003 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460453+0000", 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4851999999999999e-05 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460471+0000", 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7972e-05 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.517 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.518 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.518 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:07.518 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.994035594, 2024-09-16T04:08:07.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092583400000000003, 2024-09-16T04:08:07.518 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.518 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:07.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.37e-07 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453604+0000", 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2658e-05 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453611+0000", 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0650000000000001e-06 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453616+0000", 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9099999999999996e-06 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459323+0000", 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057072879999999996 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459369+0000", 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6436000000000001e-05 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459442+0000", 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2459000000000005e-05 2024-09-16T04:08:07.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460087+0000", 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064517999999999997 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462796+0000", 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027087639999999998 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462827+0000", 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0827000000000001e-05 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462849+0000", 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2215999999999999e-05 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.524 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.563239405000001, 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088271749999999996, 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 158 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5899999999999996e-07 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884453+0000", 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999997e-05 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884462+0000", 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2920000000000001e-06 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884468+0000", 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9399999999999999e-06 2024-09-16T04:08:07.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886225+0000", 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0017572079999999999 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886269+0000", 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3118999999999998e-05 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886334+0000", 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5695e-05 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888544+0000", 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002210086 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893152+0000", 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046078940000000004 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893177+0000", 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.463e-05 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893214+0000", 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7002000000000002e-05 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.563144376, 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087992620000000004, 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0500000000000003e-07 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884486+0000", 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-06 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884493+0000", 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8759999999999999e-06 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884498+0000", 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3550000000000004e-06 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888576+0000", 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040780230000000001 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888629+0000", 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2892000000000001e-05 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888704+0000", 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5024000000000005e-05 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892724+0000", 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040194619999999997 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893222+0000", 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049801700000000001 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893246+0000", 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3901e-05 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893281+0000", 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5277000000000001e-05 2024-09-16T04:08:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.563116341000001, 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088385849999999995, 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.537 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 160 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:07.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3900000000000005e-07 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884514+0000", 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2169999999999999e-06 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884520+0000", 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8639999999999998e-06 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884527+0000", 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.866e-06 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888915+0000", 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043874270000000002 2024-09-16T04:08:07.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888953+0000", 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7839000000000001e-05 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889014+0000", 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1289000000000002e-05 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892771+0000", 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037575199999999999 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893289+0000", 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00051718599999999999 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893314+0000", 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5202999999999999e-05 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893348+0000", 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4635000000000003e-05 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.563079160999999, 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088608530000000001, 2024-09-16T04:08:07.543 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 161 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:07.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884548+0000", 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5599999999999995e-07 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884603+0000", 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5615999999999997e-05 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884612+0000", 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6589999999999996e-06 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884619+0000", 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4630000000000004e-06 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889203+0000", 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045841989999999997 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889242+0000", 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8713e-05 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889304+0000", 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1858000000000003e-05 2024-09-16T04:08:07.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892819+0000", 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003515107 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893354+0000", 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053514000000000003 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893377+0000", 2024-09-16T04:08:07.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2265000000000001e-05 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893408+0000", 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1276999999999999e-05 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.562999912, 2024-09-16T04:08:07.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094233230000000008, 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 162 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:07.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:07.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884627+0000", 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884629+0000", 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.187e-06 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884632+0000", 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.501e-06 2024-09-16T04:08:07.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884635+0000", 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7389999999999999e-06 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892223+0000", 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075881029999999997 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892291+0000", 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7508000000000005e-05 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892412+0000", 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012124 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893979+0000", 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015676069999999999 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893987+0000", 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.306e-06 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894013+0000", 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6305999999999999e-05 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894050+0000", 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6492999999999998e-05 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.555 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.562987541, 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095013200000000006, 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.556 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 163 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2100000000000001e-07 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884642+0000", 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.478e-06 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884646+0000", 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.292e-06 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884648+0000", 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6230000000000001e-06 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892847+0000", 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0081990580000000004 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892891+0000", 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3628999999999999e-05 2024-09-16T04:08:07.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892959+0000", 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8151000000000005e-05 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894055+0000", 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001095927 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894088+0000", 2024-09-16T04:08:07.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3016999999999999e-05 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894117+0000", 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9204999999999999e-05 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894140+0000", 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2719000000000001e-05 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.562 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.562 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:07.562 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.562974003000001, 2024-09-16T04:08:07.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:07.562 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.563 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.563 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.563 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.563 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:07.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:07.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:07.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:07.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:07.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.562919942000001, 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:07.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:07.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:07.575 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:07.577 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:07.578 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.575+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:07.677 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:07.676+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:07.696 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:07.696 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:07.696 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:07.696 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:07.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:07.778 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:07.777+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:07.808 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:07.808 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:07.809 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:07.809 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:07.809 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:07.817 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:07.880 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:07.878+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:07.919 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.ds1 32/31 [PushOp(3:bc8571f8:::smithi01639990-35:head, version: 30'4, data_included: [0~143360], data_size: 143360, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bc8571f8:::smithi01639990-35:head@30'4, size: 573440, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 573440, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.681816+0000", 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.236676579, 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012026179999999999, 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:07.920 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.681816+0000", 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.681816+0000", 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.681817+0000", 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3999999999999999e-06 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.682313+0000", 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000495902 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.682390+0000", 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6947999999999996e-05 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.682412+0000", 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2514000000000002e-05 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.682525+0000", 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011231299999999999 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.683018+0000", 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00049354100000000001 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s4 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.686967+0000", 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.23152508099999999, 2024-09-16T04:08:07.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059436299999999999, 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686967+0000", 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686967+0000", 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686968+0000", 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5300000000000003e-07 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686980+0000", 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2449999999999999e-05 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687008+0000", 2024-09-16T04:08:07.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7477000000000001e-05 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687014+0000", 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7620000000000001e-06 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687079+0000", 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5539999999999999e-05 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:07.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687090+0000", 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0825000000000001e-05 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687524+0000", 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000433671 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687562+0000", 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7784999999999998e-05 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:07.928 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:07.929 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:07.929 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:07.929 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:07.929 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:07.930 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:07.981 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:07.979+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:08.032 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:08.032 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:08.033 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:08.033 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:08.041 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:08.082 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:08.081+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:08.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:08:08.178 INFO:tasks.rados.rados.0.smithi016.stdout:63: done (0 left) 2024-09-16T04:08:08.178 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 25 current snap is 2 2024-09-16T04:08:08.178 INFO:tasks.rados.rados.0.smithi016.stdout:64: seq_num 56 ranges {409600=573440,983040=376832} 2024-09-16T04:08:08.183 INFO:tasks.rados.rados.0.smithi016.stdout:64: writing smithi01639990-25 from 409600 to 983040 tid 1 2024-09-16T04:08:08.183 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:08.181+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:08.185 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:08.185 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:08.185 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:08.185 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:08.185 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:08.185 INFO:tasks.rados.rados.0.smithi016.stdout:64: writing smithi01639990-25 from 983040 to 1359872 tid 2 2024-09-16T04:08:08.186 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 16 current snap is 2 2024-09-16T04:08:08.186 INFO:tasks.rados.rados.0.smithi016.stdout:65: seq_num 57 ranges {1114112=589824,1703936=376832} 2024-09-16T04:08:08.194 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:08.195 INFO:tasks.rados.rados.0.smithi016.stdout:65: writing smithi01639990-16 from 1114112 to 1703936 tid 1 2024-09-16T04:08:08.198 INFO:tasks.rados.rados.0.smithi016.stdout:65: writing smithi01639990-16 from 1703936 to 2080768 tid 2 2024-09-16T04:08:08.198 INFO:tasks.rados.rados.0.smithi016.stdout:66: read oid 22 snap 0 2024-09-16T04:08:08.198 INFO:tasks.rados.rados.0.smithi016.stdout:66: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T04:08:08.198 INFO:tasks.rados.rados.0.smithi016.stdout:67: delete oid 40 current snap is 2 2024-09-16T04:08:08.198 INFO:tasks.rados.rados.0.smithi016.stdout:64: finishing write tid 1 to smithi01639990-25 2024-09-16T04:08:08.198 INFO:tasks.rados.rados.0.smithi016.stdout:64: finishing write tid 2 to smithi01639990-25 2024-09-16T04:08:08.198 INFO:tasks.rados.rados.0.smithi016.stdout:64: finishing write tid 3 to smithi01639990-25 2024-09-16T04:08:08.199 INFO:tasks.rados.rados.0.smithi016.stdout:64: finishing write tid 4 to smithi01639990-25 2024-09-16T04:08:08.199 INFO:tasks.rados.rados.0.smithi016.stdout:64: oid 25 updating version 0 to 8 2024-09-16T04:08:08.199 INFO:tasks.rados.rados.0.smithi016.stdout:64: oid 25 updating version 8 to 9 2024-09-16T04:08:08.199 INFO:tasks.rados.rados.0.smithi016.stdout:64: oid 25 updating version 9 to 10 2024-09-16T04:08:08.199 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 25 v 10 (ObjNum 56 snap 2 seq_num 56) dirty exists 2024-09-16T04:08:08.199 INFO:tasks.rados.rados.0.smithi016.stdout:64: left oid 25 (ObjNum 56 snap 2 seq_num 56) 2024-09-16T04:08:08.200 INFO:tasks.rados.rados.0.smithi016.stdout:66: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:65: finishing write tid 1 to smithi01639990-16 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:65: finishing write tid 2 to smithi01639990-16 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:65: finishing write tid 3 to smithi01639990-16 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:65: finishing write tid 4 to smithi01639990-16 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:65: oid 16 updating version 0 to 22 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:65: oid 16 version 22 is already newer than 21 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:65: oid 16 updating version 22 to 23 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 16 v 23 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:65: left oid 16 (ObjNum 57 snap 2 seq_num 57) 2024-09-16T04:08:08.211 INFO:tasks.rados.rados.0.smithi016.stdout:64: done (3 left) 2024-09-16T04:08:08.212 INFO:tasks.rados.rados.0.smithi016.stdout:65: done (2 left) 2024-09-16T04:08:08.212 INFO:tasks.rados.rados.0.smithi016.stdout:66: done (1 left) 2024-09-16T04:08:08.212 INFO:tasks.rados.rados.0.smithi016.stdout:67: done (0 left) 2024-09-16T04:08:08.212 INFO:tasks.rados.rados.0.smithi016.stdout:68: rollback oid 39 current snap is 2 2024-09-16T04:08:08.212 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 39 to 1 2024-09-16T04:08:08.212 INFO:tasks.rados.rados.0.smithi016.stdout:69: copy_from oid 30 from oid 16 current snap is 2 2024-09-16T04:08:08.212 INFO:tasks.rados.rados.0.smithi016.stdout:70: copy_from oid 31 from oid 48 current snap is 2 2024-09-16T04:08:08.212 INFO:tasks.rados.rados.0.smithi016.stdout:71: snap_remove snap 0 2024-09-16T04:08:08.284 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:08.282+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:08.305 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:08.305 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:08.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:08.305 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:08.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.305 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.2097211880000001, 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:08.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.ds2 32/31 ECSubRead(tid=150, to_read={3:bc8571f8:::smithi01639990-35:head=0,262144,0}, subchunks={3:bc8571f8:::smithi01639990-35:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.679991+0000", 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.62403735500000002, 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000764659, 2024-09-16T04:08:08.310 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.679991+0000", 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.679991+0000", 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.679992+0000", 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0410000000000001e-06 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.680005+0000", 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2333e-05 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.680037+0000", 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2665e-05 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.680054+0000", 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7010000000000001e-05 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.680147+0000", 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2851999999999998e-05 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.680756+0000", 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060875800000000004 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.ds4 32/31 [PushOp(3:bc8571f8:::smithi01639990-35:head, version: 30'4, data_included: [0~143360], data_size: 143360, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bc8571f8:::smithi01639990-35:head@30'4, size: 573440, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 573440, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.681940+0000", 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.62208833900000005, 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001065421, 2024-09-16T04:08:08.314 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.681940+0000", 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.681940+0000", 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.681941+0000", 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3600000000000003e-07 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.682351+0000", 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040967299999999999 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.682375+0000", 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4609000000000001e-05 2024-09-16T04:08:08.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.682391+0000", 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5909999999999998e-05 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.682472+0000", 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0501999999999996e-05 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.683006+0000", 2024-09-16T04:08:08.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053409099999999999 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.687032+0000", 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.61699698800000002, 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066545999999999999, 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687032+0000", 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687032+0000", 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687032+0000", 2024-09-16T04:08:08.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6000000000000003e-07 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687043+0000", 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0733e-05 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687071+0000", 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7555e-05 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687079+0000", 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3739999999999998e-06 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687136+0000", 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7148000000000002e-05 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687146+0000", 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6050000000000004e-06 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687665+0000", 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051899400000000005 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687697+0000", 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2391000000000002e-05 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.322 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:08.323 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:08.385 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:08.384+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:08.427 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:08.427 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:08.427 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:08.427 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:08.436 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:08.486 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:08.484+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:08.539 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:08.539 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:08.539 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:08.539 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:08.540 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:08.548 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:08.589 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:08.587+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:08.687 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:08.686+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:08.693 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:08.693 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:08.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:08.694 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:08.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.694 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s5 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:08.694 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.687081+0000", 2024-09-16T04:08:08.695 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.002857283, 2024-09-16T04:08:08.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000736234, 2024-09-16T04:08:08.695 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.695 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.695 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.695 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687081+0000", 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687081+0000", 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687082+0000", 2024-09-16T04:08:08.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.232e-06 2024-09-16T04:08:08.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687096+0000", 2024-09-16T04:08:08.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3930999999999999e-05 2024-09-16T04:08:08.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687138+0000", 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2317999999999997e-05 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687149+0000", 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0907000000000001e-05 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687221+0000", 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1958000000000005e-05 2024-09-16T04:08:08.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687233+0000", 2024-09-16T04:08:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1759e-05 2024-09-16T04:08:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687771+0000", 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053799000000000002 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.687817+0000", 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6139e-05 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.5s7 32/27 ECSubWrite(tid=34, reqid=client.4440.0:182, at_version=32'8, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.184678+0000", 2024-09-16T04:08:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.50525992900000005, 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001611765, 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.184678+0000", 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.184678+0000", 2024-09-16T04:08:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.184678+0000", 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.11e-07 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.185105+0000", 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042652400000000001 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.185131+0000", 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6687999999999999e-05 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.185140+0000", 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7630000000000006e-06 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.185211+0000", 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0718999999999998e-05 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.185223+0000", 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1736000000000001e-05 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186263+0000", 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010406090000000001 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186290+0000", 2024-09-16T04:08:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6614999999999998e-05 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.5s7 32/27 ECSubWrite(tid=35, reqid=client.4440.0:183, at_version=32'9, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.186487+0000", 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.50345128900000002, 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002194966, 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186487+0000", 2024-09-16T04:08:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186487+0000", 2024-09-16T04:08:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186488+0000", 2024-09-16T04:08:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8200000000000008e-07 2024-09-16T04:08:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186695+0000", 2024-09-16T04:08:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000207383 2024-09-16T04:08:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186722+0000", 2024-09-16T04:08:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7450999999999999e-05 2024-09-16T04:08:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186728+0000", 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5969999999999999e-06 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186781+0000", 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3322000000000002e-05 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.186787+0000", 2024-09-16T04:08:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3709999999999999e-06 2024-09-16T04:08:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.188650+0000", 2024-09-16T04:08:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001862982 2024-09-16T04:08:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.188682+0000", 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878000000000002e-05 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.5s7 32/27 ECSubWrite(tid=36, reqid=client.4440.0:184, at_version=32'10, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.187489+0000", 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.50244938100000003, 2024-09-16T04:08:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038308169999999998, 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187489+0000", 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187489+0000", 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187489+0000", 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0100000000000005e-07 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187503+0000", 2024-09-16T04:08:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3390999999999999e-05 2024-09-16T04:08:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187531+0000", 2024-09-16T04:08:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8374999999999999e-05 2024-09-16T04:08:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187537+0000", 2024-09-16T04:08:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.195e-06 2024-09-16T04:08:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187587+0000", 2024-09-16T04:08:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9784000000000003e-05 2024-09-16T04:08:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187593+0000", 2024-09-16T04:08:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2870000000000001e-06 2024-09-16T04:08:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191289+0000", 2024-09-16T04:08:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036955960000000002 2024-09-16T04:08:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191319+0000", 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0587999999999999e-05 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.5s7 32/27 ECSubWrite(tid=37, reqid=client.4440.0:184, at_version=0'0, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.191796+0000", 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.49814143, 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051553099999999995, 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191796+0000", 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191796+0000", 2024-09-16T04:08:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191797+0000", 2024-09-16T04:08:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3500000000000005e-07 2024-09-16T04:08:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191813+0000", 2024-09-16T04:08:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5404999999999999e-05 2024-09-16T04:08:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191842+0000", 2024-09-16T04:08:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9519e-05 2024-09-16T04:08:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191850+0000", 2024-09-16T04:08:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7069999999999992e-06 2024-09-16T04:08:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191908+0000", 2024-09-16T04:08:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7819000000000003e-05 2024-09-16T04:08:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191919+0000", 2024-09-16T04:08:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0793e-05 2024-09-16T04:08:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.192284+0000", 2024-09-16T04:08:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000365029 2024-09-16T04:08:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.192312+0000", 2024-09-16T04:08:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8323999999999999e-05 2024-09-16T04:08:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s7 32/27 ECSubWrite(tid=31, reqid=client.4440.0:186, at_version=32'21, trim_to=0'0, roll_forward_to=31'19))", 2024-09-16T04:08:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.196384+0000", 2024-09-16T04:08:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.49355418899999998, 2024-09-16T04:08:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003254449, 2024-09-16T04:08:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196384+0000", 2024-09-16T04:08:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196384+0000", 2024-09-16T04:08:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196384+0000", 2024-09-16T04:08:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7700000000000001e-07 2024-09-16T04:08:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196509+0000", 2024-09-16T04:08:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012490900000000001 2024-09-16T04:08:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196585+0000", 2024-09-16T04:08:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5780999999999998e-05 2024-09-16T04:08:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196633+0000", 2024-09-16T04:08:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8751000000000001e-05 2024-09-16T04:08:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196699+0000", 2024-09-16T04:08:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5623999999999994e-05 2024-09-16T04:08:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196709+0000", 2024-09-16T04:08:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0131e-05 2024-09-16T04:08:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199607+0000", 2024-09-16T04:08:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028976729999999999 2024-09-16T04:08:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199638+0000", 2024-09-16T04:08:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1303000000000002e-05 2024-09-16T04:08:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as3 32/27 ECSubWrite(tid=12, reqid=client.4440.0:193, at_version=32'8, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.197705+0000", 2024-09-16T04:08:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.49223262499999998, 2024-09-16T04:08:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027560269999999999, 2024-09-16T04:08:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197705+0000", 2024-09-16T04:08:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197705+0000", 2024-09-16T04:08:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197706+0000", 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197739+0000", 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2620999999999998e-05 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197772+0000", 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3732000000000002e-05 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197779+0000", 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2779999999999999e-06 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197845+0000", 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6427999999999995e-05 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197855+0000", 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0063e-05 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200430+0000", 2024-09-16T04:08:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025752409999999998 2024-09-16T04:08:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200461+0000", 2024-09-16T04:08:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0855999999999997e-05 2024-09-16T04:08:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s7 32/27 ECSubWrite(tid=32, reqid=client.4440.0:187, at_version=32'22, trim_to=0'0, roll_forward_to=31'19))", 2024-09-16T04:08:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.198635+0000", 2024-09-16T04:08:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.49130294499999999, 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022144389999999999, 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198635+0000", 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198635+0000", 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198636+0000", 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2200000000000002e-07 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198677+0000", 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0929000000000001e-05 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198717+0000", 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0209000000000001e-05 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198726+0000", 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0299999999999999e-06 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198800+0000", 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4369000000000006e-05 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198810+0000", 2024-09-16T04:08:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1630000000000002e-06 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200815+0000", 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002005111 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200849+0000", 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4706000000000003e-05 2024-09-16T04:08:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s7 32/27 ECSubWrite(tid=33, reqid=client.4440.0:188, at_version=32'23, trim_to=0'0, roll_forward_to=31'19))", 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.199932+0000", 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.49000580399999999, 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00094625099999999995, 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199932+0000", 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199932+0000", 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199933+0000", 2024-09-16T04:08:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3799999999999996e-07 2024-09-16T04:08:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199950+0000", 2024-09-16T04:08:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7244999999999998e-05 2024-09-16T04:08:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199995+0000", 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4515999999999999e-05 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200003+0000", 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2570000000000007e-06 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200057+0000", 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3835999999999998e-05 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200065+0000", 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3990000000000004e-06 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200855+0000", 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00079011900000000004 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200878+0000", 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3141000000000001e-05 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as3 32/27 ECSubWrite(tid=13, reqid=client.4440.0:193, at_version=0'0, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.201073+0000", 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.488864672, 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059637199999999996, 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201073+0000", 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201073+0000", 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201074+0000", 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9999999999999997e-07 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201087+0000", 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2921e-05 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201119+0000", 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2273000000000001e-05 2024-09-16T04:08:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201127+0000", 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2099999999999993e-06 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201190+0000", 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2376e-05 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201199+0000", 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8729999999999996e-06 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201640+0000", 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00044098899999999998 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201670+0000", 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0029999999999999e-05 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s7 32/27 ECSubWrite(tid=34, reqid=client.4440.0:188, at_version=0'0, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.202109+0000", 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.48782887699999999, 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000574057, 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202109+0000", 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202109+0000", 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202110+0000", 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7899999999999998e-07 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202121+0000", 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1027e-05 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202154+0000", 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2851000000000001e-05 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202162+0000", 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5120000000000008e-06 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202216+0000", 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4107000000000002e-05 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202224+0000", 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1200000000000002e-06 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202652+0000", 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000427229 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202683+0000", 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1532000000000002e-05 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 32/27 ECSubWrite(tid=24, reqid=client.4440.0:194, at_version=32'12, trim_to=0'0, roll_forward_to=31'10))", 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.210589+0000", 2024-09-16T04:08:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.479348422, 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010143699999999999, 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210589+0000", 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210589+0000", 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210590+0000", 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.28e-07 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210611+0000", 2024-09-16T04:08:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1365e-05 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210657+0000", 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5948000000000001e-05 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210666+0000", 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7749999999999998e-06 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210727+0000", 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1144999999999998e-05 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210738+0000", 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1538e-05 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211568+0000", 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00082944300000000002 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211604+0000", 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5927999999999999e-05 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 32/27 ECSubWrite(tid=25, reqid=client.4440.0:195, at_version=32'13, trim_to=0'0, roll_forward_to=31'10))", 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.211686+0000", 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.47825172100000002, 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078431099999999999, 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211686+0000", 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211686+0000", 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211687+0000", 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4600000000000004e-07 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211706+0000", 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9185e-05 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211741+0000", 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4409999999999998e-05 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211749+0000", 2024-09-16T04:08:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0160000000000008e-06 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211808+0000", 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9729999999999999e-05 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211817+0000", 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6430000000000001e-06 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212433+0000", 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000616075 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212470+0000", 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7506000000000003e-05 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 32/27 ECSubWrite(tid=26, reqid=client.4440.0:196, at_version=32'14, trim_to=0'0, roll_forward_to=31'10))", 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.212489+0000", 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.477449387, 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020125429999999999, 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212489+0000", 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212489+0000", 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212489+0000", 2024-09-16T04:08:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7199999999999998e-07 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212507+0000", 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7430000000000001e-05 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212548+0000", 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1780000000000003e-05 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212558+0000", 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9399999999999997e-06 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212618+0000", 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9366e-05 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212626+0000", 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7879999999999995e-06 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214470+0000", 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018440189999999999 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214501+0000", 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0548e-05 2024-09-16T04:08:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 32/27 ECSubWrite(tid=27, reqid=client.4440.0:196, at_version=0'0, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.214889+0000", 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.475049149, 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002931941, 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214889+0000", 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214889+0000", 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214890+0000", 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7499999999999999e-07 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214899+0000", 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7680000000000003e-06 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214936+0000", 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7039999999999998e-05 2024-09-16T04:08:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214945+0000", 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6519999999999995e-06 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215002+0000", 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7346999999999998e-05 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215012+0000", 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6919999999999997e-06 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217779+0000", 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027667550000000001 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217821+0000", 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1912000000000001e-05 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.7s4 32/27 ECSubRead(tid=76, to_read={3:e047da37:::smithi01639990-48:head=0,200704,0}, subchunks={3:e047da37:::smithi01639990-48:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.216154+0000", 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.473783393, 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043996529999999999, 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216154+0000", 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216154+0000", 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216155+0000", 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1699999999999997e-07 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216166+0000", 2024-09-16T04:08:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0966e-05 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216209+0000", 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2670999999999999e-05 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216252+0000", 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3118000000000003e-05 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216338+0000", 2024-09-16T04:08:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5654000000000004e-05 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220554+0000", 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042165270000000003 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s7 32/27 ECSubWrite(tid=11, reqid=client.4440.0:199, at_version=32'8, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.225161+0000", 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.46477658900000002, 2024-09-16T04:08:08.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044456770000000003, 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225161+0000", 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225161+0000", 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225161+0000", 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7800000000000001e-07 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225715+0000", 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00055397300000000003 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225743+0000", 2024-09-16T04:08:08.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8034999999999998e-05 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225753+0000", 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1260000000000004e-06 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225864+0000", 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011175699999999999 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225876+0000", 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1934e-05 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229540+0000", 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036639120000000001 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229607+0000", 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6761999999999995e-05 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.798 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es7 32/27 ECSubWrite(tid=28, reqid=client.4440.0:197, at_version=32'4, trim_to=0'0, roll_forward_to=30'2))", 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.227987+0000", 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.46195069599999999, 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048430519999999996, 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227987+0000", 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227987+0000", 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227989+0000", 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3650000000000001e-06 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228349+0000", 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036041099999999999 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228398+0000", 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9086999999999999e-05 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228408+0000", 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4420000000000004e-06 2024-09-16T04:08:08.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228489+0000", 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1699000000000005e-05 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228500+0000", 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0516000000000001e-05 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232804+0000", 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043043430000000004 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232830+0000", 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6188999999999999e-05 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es7 32/27 ECSubWrite(tid=29, reqid=client.4440.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=32'4))", 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.233400+0000", 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.45653792900000001, 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066326999999999998, 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233400+0000", 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233400+0000", 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233401+0000", 2024-09-16T04:08:08.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9400000000000004e-07 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233410+0000", 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9309999999999997e-06 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233439+0000", 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9116e-05 2024-09-16T04:08:08.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233448+0000", 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9730000000000003e-06 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233506+0000", 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8384000000000002e-05 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233516+0000", 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0019e-05 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.234039+0000", 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052291000000000002 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.234063+0000", 2024-09-16T04:08:08.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4142999999999999e-05 2024-09-16T04:08:08.810 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.810 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.810 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.810 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:08.810 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:08.810 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:08.812 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:08.812 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:08.810+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:08.912 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:08.911+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:08.955 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:08.956 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:08.956 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:08.956 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:08.964 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:09.048 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:09.046+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:09.089 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-16T04:08:09.092 INFO:tasks.rados.rados.0.smithi016.stdout:71: done (3 left) 2024-09-16T04:08:09.092 INFO:tasks.rados.rados.0.smithi016.stdout:72: copy_from oid 11 from oid 20 current snap is 2 2024-09-16T04:08:09.092 INFO:tasks.rados.rados.0.smithi016.stdout:68: finishing rollback tid 0 to smithi01639990-39 2024-09-16T04:08:09.092 INFO:tasks.rados.rados.0.smithi016.stdout:68: finishing rollback tid 1 to smithi01639990-39 2024-09-16T04:08:09.092 INFO:tasks.rados.rados.0.smithi016.stdout:68: finishing rollback tid 2 to smithi01639990-39 2024-09-16T04:08:09.092 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 39 v 14 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T04:08:09.092 INFO:tasks.rados.rados.0.smithi016.stdout:70: finishing copy_from racing read to smithi01639990-31 2024-09-16T04:08:09.092 INFO:tasks.rados.rados.0.smithi016.stdout:69: finishing copy_from racing read to smithi01639990-30 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:69: finishing copy_from to smithi01639990-30 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 30 v 23 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:70: finishing copy_from to smithi01639990-31 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 31 v 11 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:68: done (3 left) 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:69: done (2 left) 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:70: done (1 left) 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:73: read oid 23 snap -1 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:73: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 48 current snap is 2 2024-09-16T04:08:09.093 INFO:tasks.rados.rados.0.smithi016.stdout:74: seq_num 59 ranges {802816=622592,1425408=376832} 2024-09-16T04:08:09.102 INFO:tasks.rados.rados.0.smithi016.stdout:74: writing smithi01639990-48 from 802816 to 1425408 tid 1 2024-09-16T04:08:09.104 INFO:tasks.rados.rados.0.smithi016.stdout:74: writing smithi01639990-48 from 1425408 to 1802240 tid 2 2024-09-16T04:08:09.104 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 47 current snap is 2 2024-09-16T04:08:09.104 INFO:tasks.rados.rados.0.smithi016.stdout:75: seq_num 60 ranges {786432=638976,1425408=376832} 2024-09-16T04:08:09.113 INFO:tasks.rados.rados.0.smithi016.stdout:75: writing smithi01639990-47 from 786432 to 1425408 tid 1 2024-09-16T04:08:09.114 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:09.112+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:09.116 INFO:tasks.rados.rados.0.smithi016.stdout:75: writing smithi01639990-47 from 1425408 to 1802240 tid 2 2024-09-16T04:08:09.116 INFO:tasks.rados.rados.0.smithi016.stdout:73: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:08:09.118 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:09.118 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:09.118 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:09.118 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:09.119 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:09.129 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:72: finishing copy_from racing read to smithi01639990-11 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:74: finishing write tid 1 to smithi01639990-48 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:74: finishing write tid 2 to smithi01639990-48 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:74: finishing write tid 3 to smithi01639990-48 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:74: finishing write tid 4 to smithi01639990-48 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:74: oid 48 updating version 0 to 21 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:74: oid 48 updating version 21 to 22 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:74: oid 48 updating version 22 to 23 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 48 v 23 (ObjNum 59 snap 2 seq_num 59) dirty exists 2024-09-16T04:08:09.130 INFO:tasks.rados.rados.0.smithi016.stdout:74: left oid 48 (ObjNum 59 snap 2 seq_num 59) 2024-09-16T04:08:09.131 INFO:tasks.rados.rados.0.smithi016.stdout:72: finishing copy_from to smithi01639990-11 2024-09-16T04:08:09.131 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 11 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T04:08:09.131 INFO:tasks.rados.rados.0.smithi016.stdout:75: finishing write tid 1 to smithi01639990-47 2024-09-16T04:08:09.131 INFO:tasks.rados.rados.0.smithi016.stdout:72: done (3 left) 2024-09-16T04:08:09.131 INFO:tasks.rados.rados.0.smithi016.stdout:73: done (2 left) 2024-09-16T04:08:09.131 INFO:tasks.rados.rados.0.smithi016.stdout:74: done (1 left) 2024-09-16T04:08:09.131 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 41 current snap is 2 2024-09-16T04:08:09.131 INFO:tasks.rados.rados.0.smithi016.stdout:75: finishing write tid 2 to smithi01639990-47 2024-09-16T04:08:09.131 INFO:tasks.rados.rados.0.smithi016.stdout:76: seq_num 61 ranges {688128=655360,1343488=376832} 2024-09-16T04:08:09.136 INFO:tasks.rados.rados.0.smithi016.stdout:76: writing smithi01639990-41 from 688128 to 1343488 tid 1 2024-09-16T04:08:09.138 INFO:tasks.rados.rados.0.smithi016.stdout:76: writing smithi01639990-41 from 1343488 to 1720320 tid 2 2024-09-16T04:08:09.139 INFO:tasks.rados.rados.0.smithi016.stdout:77: snap_remove snap 1 2024-09-16T04:08:09.215 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:09.213+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:09.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:09.276 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:09.277 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:09.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:09.277 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:09.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.277 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:88 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'58, mlcod=21'54)", 2024-09-16T04:08:09.277 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.828181+0000", 2024-09-16T04:08:09.278 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.445073277999999, 2024-09-16T04:08:09.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048948150000000003, 2024-09-16T04:08:09.278 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.278 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.278 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.278 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828181+0000", 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828181+0000", 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828181+0000", 2024-09-16T04:08:09.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0399999999999996e-07 2024-09-16T04:08:09.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828185+0000", 2024-09-16T04:08:09.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.467e-06 2024-09-16T04:08:09.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828202+0000", 2024-09-16T04:08:09.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7147e-05 2024-09-16T04:08:09.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828207+0000", 2024-09-16T04:08:09.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1390000000000003e-06 2024-09-16T04:08:09.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828237+0000", 2024-09-16T04:08:09.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0386e-05 2024-09-16T04:08:09.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.828252+0000", 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4567e-05 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.833042+0000", 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047899819999999999 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.833075+0000", 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3623e-05 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:106 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'68, mlcod=21'64)", 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.998678+0000", 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.274575487, 2024-09-16T04:08:09.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0052358589999999998, 2024-09-16T04:08:09.285 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.285 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.285 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.285 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998678+0000", 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998678+0000", 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998679+0000", 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1600000000000001e-07 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998722+0000", 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2456999999999997e-05 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998748+0000", 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6679999999999999e-05 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998754+0000", 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1659999999999999e-06 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998793+0000", 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8903999999999999e-05 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.998808+0000", 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4724e-05 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003876+0000", 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050678140000000003 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003914+0000", 2024-09-16T04:08:09.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8398000000000001e-05 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:107 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'69, mlcod=21'64)", 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.999067+0000", 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.274187414, 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048815799999999999, 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999067+0000", 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999067+0000", 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999067+0000", 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4600000000000004e-07 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999073+0000", 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8409999999999996e-06 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999099+0000", 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5936e-05 2024-09-16T04:08:09.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999105+0000", 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7509999999999997e-06 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999139+0000", 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4230000000000003e-05 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999152+0000", 2024-09-16T04:08:09.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2713e-05 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003917+0000", 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004765516 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003948+0000", 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0846999999999998e-05 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.296 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.297 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:108 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'70, mlcod=21'64)", 2024-09-16T04:08:09.297 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.999445+0000", 2024-09-16T04:08:09.298 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.273809281999998, 2024-09-16T04:08:09.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045202180000000003, 2024-09-16T04:08:09.298 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.298 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.298 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.298 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999445+0000", 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999445+0000", 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999449+0000", 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4109999999999998e-06 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999454+0000", 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7670000000000003e-06 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999474+0000", 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0111000000000001e-05 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999479+0000", 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2560000000000002e-06 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999519+0000", 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9672999999999998e-05 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999532+0000", 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3040999999999999e-05 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003949+0000", 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044173839999999999 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003965+0000", 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5574999999999999e-05 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:109 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'71, mlcod=21'65)", 2024-09-16T04:08:09.302 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:51.999872+0000", 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.273381965999999, 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041208269999999996, 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999872+0000", 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999872+0000", 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999872+0000", 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1699999999999999e-07 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999940+0000", 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7838999999999999e-05 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999960+0000", 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0134999999999999e-05 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:51.999966+0000", 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5230000000000003e-06 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000005+0000", 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8994e-05 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000021+0000", 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6192e-05 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003967+0000", 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039464169999999998 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003993+0000", 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.531e-05 2024-09-16T04:08:09.306 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:110 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'72, mlcod=21'65)", 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.000242+0000", 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.273012418, 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037826499999999998, 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000242+0000", 2024-09-16T04:08:09.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000242+0000", 2024-09-16T04:08:09.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000242+0000", 2024-09-16T04:08:09.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2699999999999998e-07 2024-09-16T04:08:09.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000248+0000", 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9440000000000002e-06 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000273+0000", 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4593999999999999e-05 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000278+0000", 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4399999999999996e-06 2024-09-16T04:08:09.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000315+0000", 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7073999999999998e-05 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000328+0000", 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2879e-05 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.003994+0000", 2024-09-16T04:08:09.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003665705 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004024+0000", 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0187000000000001e-05 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:111 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'73, mlcod=21'65)", 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.000585+0000", 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.272669304000001, 2024-09-16T04:08:09.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034651180000000001, 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000585+0000", 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000585+0000", 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000585+0000", 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1399999999999997e-07 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000589+0000", 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0249999999999996e-06 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000608+0000", 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8479e-05 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000614+0000", 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9179999999999997e-06 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000666+0000", 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1103000000000002e-05 2024-09-16T04:08:09.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000678+0000", 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2775999999999999e-05 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004026+0000", 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0033480960000000001 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004050+0000", 2024-09-16T04:08:09.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3306999999999999e-05 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:112 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'74, mlcod=21'65)", 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.000915+0000", 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.272338971, 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003398014, 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000915+0000", 2024-09-16T04:08:09.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000915+0000", 2024-09-16T04:08:09.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000916+0000", 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6899999999999997e-07 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.000989+0000", 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3436999999999995e-05 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001010+0000", 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1248999999999999e-05 2024-09-16T04:08:09.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001016+0000", 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8429999999999998e-06 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001066+0000", 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0247000000000002e-05 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.001082+0000", 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5960999999999998e-05 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004290+0000", 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032079360000000002 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.004313+0000", 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2671999999999999e-05 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:124 1.0 e22/19 1:30306672:devicehealth::main.db.0000000000000000:head v 22'86, mlcod=21'84)", 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.842472711999999, 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020095396000000001, 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.323 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:09.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430782+0000", 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9699999999999996e-07 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430789+0000", 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4660000000000002e-06 2024-09-16T04:08:09.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450028+0000", 2024-09-16T04:08:09.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019238816999999998 2024-09-16T04:08:09.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450042+0000", 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4487e-05 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450169+0000", 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126787 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450188+0000", 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:08:09.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450816+0000", 2024-09-16T04:08:09.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062741399999999997 2024-09-16T04:08:09.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450877+0000", 2024-09-16T04:08:09.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0782999999999997e-05 2024-09-16T04:08:09.329 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.329 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.329 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.5s6 32/27 ECSubWrite(tid=36, reqid=client.4440.0:184, at_version=32'10, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.187518+0000", 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.085735672, 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038288189999999998, 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187518+0000", 2024-09-16T04:08:09.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187518+0000", 2024-09-16T04:08:09.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187519+0000", 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0999999999999998e-07 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187534+0000", 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5089000000000001e-05 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187559+0000", 2024-09-16T04:08:09.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4749999999999999e-05 2024-09-16T04:08:09.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187568+0000", 2024-09-16T04:08:09.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8599999999999999e-06 2024-09-16T04:08:09.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187606+0000", 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8009999999999997e-05 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.187611+0000", 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9969999999999998e-06 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191310+0000", 2024-09-16T04:08:09.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036995880000000002 2024-09-16T04:08:09.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.191347+0000", 2024-09-16T04:08:09.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6915000000000001e-05 2024-09-16T04:08:09.335 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.335 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:194 3.8s0 3:17d1552f:::smithi01639990-39:head [append 0~0] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209870+0000", 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0633842520000001, 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003208289, 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.336 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 194 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209870+0000", 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209870+0000", 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209872+0000", 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2259999999999999e-06 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209880+0000", 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3149999999999992e-06 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209897+0000", 2024-09-16T04:08:09.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6631999999999999e-05 2024-09-16T04:08:09.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209905+0000", 2024-09-16T04:08:09.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3289999999999994e-06 2024-09-16T04:08:09.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209967+0000", 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2106999999999993e-05 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210033+0000", 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5759999999999994e-05 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210539+0000", 2024-09-16T04:08:09.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050642000000000005 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212661+0000", 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021210769999999999 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.213012+0000", 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000351226 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.213078+0000", 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6197000000000003e-05 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:195 3.8s0 3:17d1552f:::smithi01639990-39:head [rollback 3] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209928+0000", 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.063325879, 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035400309999999999, 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.342 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 195 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209928+0000", 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209928+0000", 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209929+0000", 2024-09-16T04:08:09.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4600000000000005e-07 2024-09-16T04:08:09.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209931+0000", 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6429999999999999e-06 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209947+0000", 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6002e-05 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7090000000000008e-06 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211034+0000", 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010778700000000001 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211114+0000", 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9764000000000004e-05 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211653+0000", 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053981500000000002 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212705+0000", 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010517440000000001 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.213423+0000", 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071764799999999998 2024-09-16T04:08:09.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.213468+0000", 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5290000000000002e-05 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:196 3.8s0 3:17d1552f:::smithi01639990-39:head [append 0~0] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209967+0000", 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0632873309999999, 2024-09-16T04:08:09.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047564499999999997, 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 196 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209967+0000", 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209967+0000", 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209967+0000", 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5200000000000002e-07 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209970+0000", 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.481e-06 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209976+0000", 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9489999999999999e-06 2024-09-16T04:08:09.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209983+0000", 2024-09-16T04:08:09.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2559999999999999e-06 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.211950+0000", 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0019674010000000001 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212033+0000", 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2948000000000006e-05 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.212459+0000", 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042571900000000002 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.213253+0000", 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00079443500000000002 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214678+0000", 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014250479999999999 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214723+0000", 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4761000000000001e-05 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:197 3.es0 3:7f241f48:::smithi01639990-30:head [copy-from ver 23] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.063263364, 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.023242381999999999, 2024-09-16T04:08:09.355 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 197 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:09.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:09.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6499999999999999e-07 2024-09-16T04:08:09.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209995+0000", 2024-09-16T04:08:09.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.388e-06 2024-09-16T04:08:09.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210002+0000", 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5209999999999997e-06 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210008+0000", 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8780000000000003e-06 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:09.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8421999999999999e-05 2024-09-16T04:08:09.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210099+0000", 2024-09-16T04:08:09.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2469999999999997e-05 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227450+0000", 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017351235 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232352+0000", 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049017239999999997 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233168+0000", 2024-09-16T04:08:09.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081616600000000001 2024-09-16T04:08:09.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.362 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233233+0000", 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5012999999999996e-05 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:198 3.es0 3:7f241f48:::smithi01639990-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.063233055, 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019673758, 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.363 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 198 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3099999999999998e-07 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210024+0000", 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2620000000000001e-06 2024-09-16T04:08:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210030+0000", 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9020000000000002e-06 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210037+0000", 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1480000000000003e-06 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210227+0000", 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00019055499999999999 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210269+0000", 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2246999999999997e-05 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229480+0000", 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019210896000000002 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229587+0000", 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107026 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229695+0000", 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107291 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.368 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s4 33/27 ECSubWrite(tid=13, reqid=osd.4.0:13, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.096186+0000", 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.177068428, 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037093059999999999, 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096186+0000", 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096186+0000", 2024-09-16T04:08:09.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096186+0000", 2024-09-16T04:08:09.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.05e-07 2024-09-16T04:08:09.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096205+0000", 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9273999999999999e-05 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096265+0000", 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9787000000000002e-05 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096272+0000", 2024-09-16T04:08:09.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.43e-06 2024-09-16T04:08:09.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096330+0000", 2024-09-16T04:08:09.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7655000000000003e-05 2024-09-16T04:08:09.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096340+0000", 2024-09-16T04:08:09.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7310000000000005e-06 2024-09-16T04:08:09.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099867+0000", 2024-09-16T04:08:09.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035274920000000001 2024-09-16T04:08:09.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099895+0000", 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7732000000000001e-05 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s1 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.100031+0000", 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.173223183, 2024-09-16T04:08:09.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031784389999999999, 2024-09-16T04:08:09.375 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.375 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100031+0000", 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100031+0000", 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100031+0000", 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.37e-07 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100052+0000", 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0343000000000002e-05 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100098+0000", 2024-09-16T04:08:09.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.583e-05 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100105+0000", 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.79e-06 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100418+0000", 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000312971 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100426+0000", 2024-09-16T04:08:09.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.311e-06 2024-09-16T04:08:09.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103181+0000", 2024-09-16T04:08:09.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027557860000000001 2024-09-16T04:08:09.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103209+0000", 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7770999999999999e-05 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s1 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.107505+0000", 2024-09-16T04:08:09.380 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.16574908599999999, 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0033933560000000001, 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107505+0000", 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107505+0000", 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107505+0000", 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.22e-07 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107718+0000", 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021312999999999999 2024-09-16T04:08:09.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107735+0000", 2024-09-16T04:08:09.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6971000000000001e-05 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107740+0000", 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4700000000000004e-06 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107790+0000", 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0671000000000003e-05 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107798+0000", 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9170000000000006e-06 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110867+0000", 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030685069999999998 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110898+0000", 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1368000000000002e-05 2024-09-16T04:08:09.385 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:211 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~638976] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.112759+0000", 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.160495165, 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011041176, 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:09.386 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 211 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.112759+0000", 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.112759+0000", 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.112761+0000", 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.872e-06 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.113913+0000", 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001151946 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.113925+0000", 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2102e-05 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.113944+0000", 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8913999999999999e-05 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.114003+0000", 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9398000000000003e-05 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.114090+0000", 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6681999999999994e-05 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115489+0000", 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001399335 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120457+0000", 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049679219999999996 2024-09-16T04:08:09.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123743+0000", 2024-09-16T04:08:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032862799999999999 2024-09-16T04:08:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123800+0000", 2024-09-16T04:08:09.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6725000000000003e-05 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:212 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~376832] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.115771+0000", 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.157483395, 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010791148, 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:09.392 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 212 2024-09-16T04:08:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:09.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115771+0000", 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115771+0000", 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:09.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115772+0000", 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.669e-06 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116062+0000", 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000289456 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116073+0000", 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1612e-05 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116081+0000", 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.117461+0000", 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001379882 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.117566+0000", 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105175 2024-09-16T04:08:09.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118399+0000", 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00083219399999999997 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122364+0000", 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003965016 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126483+0000", 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041189030000000001 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126562+0000", 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9141999999999995e-05 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.398 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:09.399 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:09.400 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:09.400 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:09.401 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:09.402 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:09.398+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:09.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:09.407 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:09.428 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:09.428 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:09.450 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:09.472 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:09.494 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:09.501 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:09.499+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:09.510 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:09.510 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:09.516 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:09.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:09.538 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:09.538 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:09.560 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:09.604 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:09.600+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:09.645 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:09.646 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:09.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:09.702 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:09.701+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:09.707 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:09.729 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:09.803 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:09.801+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:09.804 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:09.804 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:09.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:08:09.905 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:09.903+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:09.960 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:09.960 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:09.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:08:10.014 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:10.012+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:10.077 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:10.077 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:10.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:08:10.095 INFO:tasks.rados.rados.0.smithi016.stdout:77: done (2 left) 2024-09-16T04:08:10.095 INFO:tasks.rados.rados.0.smithi016.stdout:78: read oid 23 snap -1 2024-09-16T04:08:10.095 INFO:tasks.rados.rados.0.smithi016.stdout:78: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:08:10.095 INFO:tasks.rados.rados.0.smithi016.stdout:79: delete oid 10 current snap is 2 2024-09-16T04:08:10.095 INFO:tasks.rados.rados.0.smithi016.stdout:76: finishing write tid 1 to smithi01639990-41 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:76: finishing write tid 2 to smithi01639990-41 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:75: finishing write tid 3 to smithi01639990-47 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:75: finishing write tid 4 to smithi01639990-47 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:75: oid 47 updating version 0 to 20 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:75: oid 47 updating version 20 to 22 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:75: oid 47 version 22 is already newer than 21 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 47 v 22 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:75: left oid 47 (ObjNum 60 snap 2 seq_num 60) 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:76: finishing write tid 3 to smithi01639990-41 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:76: finishing write tid 4 to smithi01639990-41 2024-09-16T04:08:10.096 INFO:tasks.rados.rados.0.smithi016.stdout:76: oid 41 updating version 0 to 16 2024-09-16T04:08:10.097 INFO:tasks.rados.rados.0.smithi016.stdout:76: oid 41 version 16 is already newer than 14 2024-09-16T04:08:10.097 INFO:tasks.rados.rados.0.smithi016.stdout:76: oid 41 version 16 is already newer than 15 2024-09-16T04:08:10.097 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 41 v 16 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-09-16T04:08:10.097 INFO:tasks.rados.rados.0.smithi016.stdout:76: left oid 41 (ObjNum 61 snap 2 seq_num 61) 2024-09-16T04:08:10.103 INFO:tasks.rados.rados.0.smithi016.stdout:78: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T04:08:10.106 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:10.104+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:10.119 INFO:tasks.rados.rados.0.smithi016.stdout:75: done (3 left) 2024-09-16T04:08:10.119 INFO:tasks.rados.rados.0.smithi016.stdout:76: done (2 left) 2024-09-16T04:08:10.120 INFO:tasks.rados.rados.0.smithi016.stdout:78: done (1 left) 2024-09-16T04:08:10.120 INFO:tasks.rados.rados.0.smithi016.stdout:79: done (0 left) 2024-09-16T04:08:10.120 INFO:tasks.rados.rados.0.smithi016.stdout:80: rmattr oid 5 current snap is 2 2024-09-16T04:08:10.120 INFO:tasks.rados.rados.0.smithi016.stdout:80: done (0 left) 2024-09-16T04:08:10.120 INFO:tasks.rados.rados.0.smithi016.stdout:81: setattr oid 45 current snap is 2 2024-09-16T04:08:10.120 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 1 current snap is 2 2024-09-16T04:08:10.121 INFO:tasks.rados.rados.0.smithi016.stdout:82: seq_num 63 ranges {32768=688128,720896=376832} 2024-09-16T04:08:10.125 INFO:tasks.rados.rados.0.smithi016.stdout:82: writing smithi01639990-1 from 32768 to 720896 tid 1 2024-09-16T04:08:10.127 INFO:tasks.rados.rados.0.smithi016.stdout:82: writing smithi01639990-1 from 720896 to 1097728 tid 2 2024-09-16T04:08:10.128 INFO:tasks.rados.rados.0.smithi016.stdout:83: rollback oid 13 current snap is 2 2024-09-16T04:08:10.128 INFO:tasks.rados.rados.0.smithi016.stdout:83: done (2 left) 2024-09-16T04:08:10.128 INFO:tasks.rados.rados.0.smithi016.stdout:84: copy_from oid 48 from oid 34 current snap is 2 2024-09-16T04:08:10.128 INFO:tasks.rados.rados.0.smithi016.stdout:85: read oid 29 snap -1 2024-09-16T04:08:10.128 INFO:tasks.rados.rados.0.smithi016.stdout:85: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T04:08:10.128 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 7 current snap is 2 2024-09-16T04:08:10.128 INFO:tasks.rados.rados.0.smithi016.stdout:86: seq_num 64 ranges {114688=704512,819200=376832} 2024-09-16T04:08:10.132 INFO:tasks.rados.rados.0.smithi016.stdout:86: writing smithi01639990-7 from 114688 to 819200 tid 1 2024-09-16T04:08:10.135 INFO:tasks.rados.rados.0.smithi016.stdout:86: writing smithi01639990-7 from 819200 to 1196032 tid 2 2024-09-16T04:08:10.135 INFO:tasks.rados.rados.0.smithi016.stdout:87: setattr oid 11 current snap is 2 2024-09-16T04:08:10.136 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T04:08:10.136 INFO:tasks.rados.rados.0.smithi016.stdout:82: finishing write tid 1 to smithi01639990-1 2024-09-16T04:08:10.136 INFO:tasks.rados.rados.0.smithi016.stdout:82: finishing write tid 2 to smithi01639990-1 2024-09-16T04:08:10.136 INFO:tasks.rados.rados.0.smithi016.stdout:82: finishing write tid 3 to smithi01639990-1 2024-09-16T04:08:10.137 INFO:tasks.rados.rados.0.smithi016.stdout:82: finishing write tid 4 to smithi01639990-1 2024-09-16T04:08:10.137 INFO:tasks.rados.rados.0.smithi016.stdout:82: oid 1 updating version 0 to 17 2024-09-16T04:08:10.137 INFO:tasks.rados.rados.0.smithi016.stdout:82: oid 1 version 17 is already newer than 16 2024-09-16T04:08:10.138 INFO:tasks.rados.rados.0.smithi016.stdout:82: oid 1 version 17 is already newer than 15 2024-09-16T04:08:10.138 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 1 v 17 (ObjNum 63 snap 2 seq_num 63) dirty exists 2024-09-16T04:08:10.138 INFO:tasks.rados.rados.0.smithi016.stdout:82: left oid 1 (ObjNum 63 snap 2 seq_num 63) 2024-09-16T04:08:10.138 INFO:tasks.rados.rados.0.smithi016.stdout:81: done (5 left) 2024-09-16T04:08:10.138 INFO:tasks.rados.rados.0.smithi016.stdout:82: done (4 left) 2024-09-16T04:08:10.138 INFO:tasks.rados.rados.0.smithi016.stdout:88: snap_create 2024-09-16T04:08:10.138 INFO:tasks.rados.rados.0.smithi016.stdout:85: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T04:08:10.154 INFO:tasks.rados.rados.0.smithi016.stdout:86: finishing write tid 1 to smithi01639990-7 2024-09-16T04:08:10.154 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 11 v 32 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T04:08:10.154 INFO:tasks.rados.rados.0.smithi016.stdout:86: finishing write tid 2 to smithi01639990-7 2024-09-16T04:08:10.154 INFO:tasks.rados.rados.0.smithi016.stdout:84: finishing copy_from racing read to smithi01639990-48 2024-09-16T04:08:10.164 INFO:tasks.rados.rados.0.smithi016.stdout:84: finishing copy_from to smithi01639990-48 2024-09-16T04:08:10.164 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 48 v 19 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-09-16T04:08:10.206 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:10.205+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:10.224 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:10.224 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:10.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:10.308 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:10.306+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:10.380 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:10.381 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:10.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:10.409 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:10.407+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:10.496 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:10.496 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:10.505 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:10.509 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:10.507+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:10.610 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:10.608+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:10.612 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:10.612 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:10.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:10.711 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:10.710+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:10.728 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:10.728 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:10.737 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:10.812 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:10.811+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:10.844 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:10.844 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:10.853 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:10.913 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:10.911+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:10.946 INFO:tasks.rados.rados.0.smithi016.stdout:86: finishing write tid 3 to smithi01639990-7 2024-09-16T04:08:10.947 INFO:tasks.rados.rados.0.smithi016.stdout:86: finishing write tid 4 to smithi01639990-7 2024-09-16T04:08:10.947 INFO:tasks.rados.rados.0.smithi016.stdout:86: oid 7 updating version 0 to 11 2024-09-16T04:08:10.947 INFO:tasks.rados.rados.0.smithi016.stdout:86: oid 7 updating version 11 to 12 2024-09-16T04:08:10.947 INFO:tasks.rados.rados.0.smithi016.stdout:86: oid 7 updating version 12 to 13 2024-09-16T04:08:10.947 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 7 v 13 (ObjNum 64 snap 2 seq_num 64) dirty exists 2024-09-16T04:08:10.947 INFO:tasks.rados.rados.0.smithi016.stdout:86: left oid 7 (ObjNum 64 snap 2 seq_num 64) 2024-09-16T04:08:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:10.961 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:10.970 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:11.014 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:11.013+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:11.086 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:11.087 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:11.096 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:11.097 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T04:08:11.097 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:08:11.097 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-16T04:08:11.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 3.0 2024-09-16T04:08:11.100 INFO:tasks.rados.rados.0.smithi016.stdout:84: done (4 left) 2024-09-16T04:08:11.101 INFO:tasks.rados.rados.0.smithi016.stdout:85: done (3 left) 2024-09-16T04:08:11.101 INFO:tasks.rados.rados.0.smithi016.stdout:86: done (2 left) 2024-09-16T04:08:11.101 INFO:tasks.rados.rados.0.smithi016.stdout:87: done (1 left) 2024-09-16T04:08:11.101 INFO:tasks.rados.rados.0.smithi016.stdout:88: done (0 left) 2024-09-16T04:08:11.101 INFO:tasks.rados.rados.0.smithi016.stdout:89: rmattr oid 39 current snap is 3 2024-09-16T04:08:11.101 INFO:tasks.rados.rados.0.smithi016.stdout:89: done (0 left) 2024-09-16T04:08:11.101 INFO:tasks.rados.rados.0.smithi016.stdout:90: delete oid 4 current snap is 3 2024-09-16T04:08:11.113 INFO:tasks.rados.rados.0.smithi016.stdout:90: done (0 left) 2024-09-16T04:08:11.113 INFO:tasks.rados.rados.0.smithi016.stdout:91: rollback oid 5 current snap is 3 2024-09-16T04:08:11.113 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 5 to 2 2024-09-16T04:08:11.113 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 20 current snap is 3 2024-09-16T04:08:11.113 INFO:tasks.rados.rados.0.smithi016.stdout:92: seq_num 66 ranges {327680=737280,1064960=376832} 2024-09-16T04:08:11.115 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:11.114+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:11.119 INFO:tasks.rados.rados.0.smithi016.stdout:92: writing smithi01639990-20 from 327680 to 1064960 tid 1 2024-09-16T04:08:11.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:08:11.122 INFO:tasks.rados.rados.0.smithi016.stdout:92: writing smithi01639990-20 from 1064960 to 1441792 tid 2 2024-09-16T04:08:11.122 INFO:tasks.rados.rados.0.smithi016.stdout:93: setattr oid 4 current snap is 3 2024-09-16T04:08:11.122 INFO:tasks.rados.rados.0.smithi016.stdout:91: finishing rollback tid 0 to smithi01639990-5 2024-09-16T04:08:11.122 INFO:tasks.rados.rados.0.smithi016.stdout:91: finishing rollback tid 1 to smithi01639990-5 2024-09-16T04:08:11.122 INFO:tasks.rados.rados.0.smithi016.stdout:91: finishing rollback tid 2 to smithi01639990-5 2024-09-16T04:08:11.122 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 5 v 8 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:08:11.125 INFO:tasks.rados.rados.0.smithi016.stdout:91: done (2 left) 2024-09-16T04:08:11.125 INFO:tasks.rados.rados.0.smithi016.stdout:94: rollback oid 11 current snap is 3 2024-09-16T04:08:11.125 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 11 to 2 2024-09-16T04:08:11.125 INFO:tasks.rados.rados.0.smithi016.stdout:95: copy_from oid 31 from oid 38 current snap is 3 2024-09-16T04:08:11.125 INFO:tasks.rados.rados.0.smithi016.stdout:96: delete oid 23 current snap is 3 2024-09-16T04:08:11.129 INFO:tasks.rados.rados.0.smithi016.stdout:92: finishing write tid 1 to smithi01639990-20 2024-09-16T04:08:11.131 INFO:tasks.rados.rados.0.smithi016.stdout:92: finishing write tid 2 to smithi01639990-20 2024-09-16T04:08:11.131 INFO:tasks.rados.rados.0.smithi016.stdout:94: finishing rollback tid 0 to smithi01639990-11 2024-09-16T04:08:11.131 INFO:tasks.rados.rados.0.smithi016.stdout:92: finishing write tid 3 to smithi01639990-20 2024-09-16T04:08:11.132 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 4 v 17 (ObjNum 129 snap 0 seq_num 1558714072) dirty exists 2024-09-16T04:08:11.132 INFO:tasks.rados.rados.0.smithi016.stdout:93: done (4 left) 2024-09-16T04:08:11.132 INFO:tasks.rados.rados.0.smithi016.stdout:96: done (3 left) 2024-09-16T04:08:11.132 INFO:tasks.rados.rados.0.smithi016.stdout:97: delete oid 40 current snap is 3 2024-09-16T04:08:11.132 INFO:tasks.rados.rados.0.smithi016.stdout:94: finishing rollback tid 1 to smithi01639990-11 2024-09-16T04:08:11.133 INFO:tasks.rados.rados.0.smithi016.stdout:92: finishing write tid 4 to smithi01639990-20 2024-09-16T04:08:11.133 INFO:tasks.rados.rados.0.smithi016.stdout:92: oid 20 updating version 0 to 15 2024-09-16T04:08:11.133 INFO:tasks.rados.rados.0.smithi016.stdout:92: oid 20 version 15 is already newer than 14 2024-09-16T04:08:11.133 INFO:tasks.rados.rados.0.smithi016.stdout:92: oid 20 updating version 15 to 16 2024-09-16T04:08:11.133 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 20 v 16 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-09-16T04:08:11.133 INFO:tasks.rados.rados.0.smithi016.stdout:92: left oid 20 (ObjNum 66 snap 3 seq_num 66) 2024-09-16T04:08:11.135 INFO:tasks.rados.rados.0.smithi016.stdout:92: done (3 left) 2024-09-16T04:08:11.135 INFO:tasks.rados.rados.0.smithi016.stdout:97: done (2 left) 2024-09-16T04:08:11.135 INFO:tasks.rados.rados.0.smithi016.stdout:98: setattr oid 41 current snap is 3 2024-09-16T04:08:11.135 INFO:tasks.rados.rados.0.smithi016.stdout:95: finishing copy_from racing read to smithi01639990-31 2024-09-16T04:08:11.136 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 35 current snap is 3 2024-09-16T04:08:11.136 INFO:tasks.rados.rados.0.smithi016.stdout:99: seq_num 69 ranges {573440=786432,1359872=376832} 2024-09-16T04:08:11.144 INFO:tasks.rados.rados.0.smithi016.stdout:99: writing smithi01639990-35 from 573440 to 1359872 tid 1 2024-09-16T04:08:11.147 INFO:tasks.rados.rados.0.smithi016.stdout:99: writing smithi01639990-35 from 1359872 to 1736704 tid 2 2024-09-16T04:08:11.147 INFO:tasks.rados.rados.0.smithi016.stdout:100: read oid 33 snap -1 2024-09-16T04:08:11.147 INFO:tasks.rados.rados.0.smithi016.stdout:100: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:08:11.147 INFO:tasks.rados.rados.0.smithi016.stdout:101: copy_from oid 45 from oid 17 current snap is 3 2024-09-16T04:08:11.147 INFO:tasks.rados.rados.0.smithi016.stdout:95: finishing copy_from to smithi01639990-31 2024-09-16T04:08:11.147 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 31 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T04:08:11.147 INFO:tasks.rados.rados.0.smithi016.stdout:95: done (5 left) 2024-09-16T04:08:11.148 INFO:tasks.rados.rados.0.smithi016.stdout:102: rmattr oid 20 current snap is 3 2024-09-16T04:08:11.148 INFO:tasks.rados.rados.0.smithi016.stdout:102: done (5 left) 2024-09-16T04:08:11.148 INFO:tasks.rados.rados.0.smithi016.stdout:103: delete oid 20 current snap is 3 2024-09-16T04:08:11.148 INFO:tasks.rados.rados.0.smithi016.stdout:94: finishing rollback tid 2 to smithi01639990-11 2024-09-16T04:08:11.148 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 11 v 37 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T04:08:11.148 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 41 v 20 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-09-16T04:08:11.152 INFO:tasks.rados.rados.0.smithi016.stdout:99: finishing write tid 1 to smithi01639990-35 2024-09-16T04:08:11.157 INFO:tasks.rados.rados.0.smithi016.stdout:99: finishing write tid 2 to smithi01639990-35 2024-09-16T04:08:11.158 INFO:tasks.rados.rados.0.smithi016.stdout:99: finishing write tid 3 to smithi01639990-35 2024-09-16T04:08:11.158 INFO:tasks.rados.rados.0.smithi016.stdout:101: finishing copy_from racing read to smithi01639990-45 2024-09-16T04:08:11.160 INFO:tasks.rados.rados.0.smithi016.stdout:99: finishing write tid 4 to smithi01639990-35 2024-09-16T04:08:11.160 INFO:tasks.rados.rados.0.smithi016.stdout:99: oid 35 updating version 0 to 11 2024-09-16T04:08:11.160 INFO:tasks.rados.rados.0.smithi016.stdout:99: oid 35 version 11 is already newer than 10 2024-09-16T04:08:11.160 INFO:tasks.rados.rados.0.smithi016.stdout:99: oid 35 updating version 11 to 12 2024-09-16T04:08:11.160 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 35 v 12 (ObjNum 69 snap 3 seq_num 69) dirty exists 2024-09-16T04:08:11.161 INFO:tasks.rados.rados.0.smithi016.stdout:99: left oid 35 (ObjNum 69 snap 3 seq_num 69) 2024-09-16T04:08:11.161 INFO:tasks.rados.rados.0.smithi016.stdout:101: finishing copy_from to smithi01639990-45 2024-09-16T04:08:11.162 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 45 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T04:08:11.162 INFO:tasks.rados.rados.0.smithi016.stdout:94: done (5 left) 2024-09-16T04:08:11.162 INFO:tasks.rados.rados.0.smithi016.stdout:98: done (4 left) 2024-09-16T04:08:11.162 INFO:tasks.rados.rados.0.smithi016.stdout:99: done (3 left) 2024-09-16T04:08:11.162 INFO:tasks.rados.rados.0.smithi016.stdout:101: done (2 left) 2024-09-16T04:08:11.162 INFO:tasks.rados.rados.0.smithi016.stdout:103: done (1 left) 2024-09-16T04:08:11.162 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 11 current snap is 3 2024-09-16T04:08:11.162 INFO:tasks.rados.rados.0.smithi016.stdout:104: seq_num 71 ranges {327680=425984,753664=573440,1327104=196608} 2024-09-16T04:08:11.165 INFO:tasks.rados.rados.0.smithi016.stdout:104: writing smithi01639990-11 from 327680 to 753664 tid 1 2024-09-16T04:08:11.169 INFO:tasks.rados.rados.0.smithi016.stdout:104: writing smithi01639990-11 from 753664 to 1327104 tid 2 2024-09-16T04:08:11.171 INFO:tasks.rados.rados.0.smithi016.stdout:104: writing smithi01639990-11 from 1327104 to 1523712 tid 3 2024-09-16T04:08:11.171 INFO:tasks.rados.rados.0.smithi016.stdout:105: read oid 24 snap -1 2024-09-16T04:08:11.171 INFO:tasks.rados.rados.0.smithi016.stdout:105: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T04:08:11.171 INFO:tasks.rados.rados.0.smithi016.stdout:106: rollback oid 44 current snap is 3 2024-09-16T04:08:11.171 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 44 to 2 2024-09-16T04:08:11.171 INFO:tasks.rados.rados.0.smithi016.stdout:107: read oid 22 snap -1 2024-09-16T04:08:11.171 INFO:tasks.rados.rados.0.smithi016.stdout:107: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T04:08:11.171 INFO:tasks.rados.rados.0.smithi016.stdout:108: copy_from oid 6 from oid 19 current snap is 3 2024-09-16T04:08:11.172 INFO:tasks.rados.rados.0.smithi016.stdout:109: delete oid 36 current snap is 3 2024-09-16T04:08:11.175 INFO:tasks.rados.rados.0.smithi016.stdout:104: finishing write tid 1 to smithi01639990-11 2024-09-16T04:08:11.176 INFO:tasks.rados.rados.0.smithi016.stdout:104: finishing write tid 2 to smithi01639990-11 2024-09-16T04:08:11.177 INFO:tasks.rados.rados.0.smithi016.stdout:106: finishing rollback tid 0 to smithi01639990-44 2024-09-16T04:08:11.178 INFO:tasks.rados.rados.0.smithi016.stdout:108: finishing copy_from to smithi01639990-6 2024-09-16T04:08:11.178 INFO:tasks.rados.rados.0.smithi016.stdout:108: got expected ENOENT (src dne) 2024-09-16T04:08:11.178 INFO:tasks.rados.rados.0.smithi016.stdout:108: finishing copy_from racing read to smithi01639990-6 2024-09-16T04:08:11.180 INFO:tasks.rados.rados.0.smithi016.stdout:105: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T04:08:11.199 INFO:tasks.rados.rados.0.smithi016.stdout:104: finishing write tid 3 to smithi01639990-11 2024-09-16T04:08:11.199 INFO:tasks.rados.rados.0.smithi016.stdout:107: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T04:08:11.213 INFO:tasks.rados.rados.0.smithi016.stdout:106: finishing rollback tid 1 to smithi01639990-44 2024-09-16T04:08:11.213 INFO:tasks.rados.rados.0.smithi016.stdout:106: finishing rollback tid 2 to smithi01639990-44 2024-09-16T04:08:11.213 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 44 v 15 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T04:08:11.217 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:11.215+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:11.218 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:11.218 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:11.222 INFO:tasks.rados.rados.0.smithi016.stdout:105: done (6 left) 2024-09-16T04:08:11.222 INFO:tasks.rados.rados.0.smithi016.stdout:106: done (5 left) 2024-09-16T04:08:11.222 INFO:tasks.rados.rados.0.smithi016.stdout:107: done (4 left) 2024-09-16T04:08:11.222 INFO:tasks.rados.rados.0.smithi016.stdout:108: done (3 left) 2024-09-16T04:08:11.222 INFO:tasks.rados.rados.0.smithi016.stdout:109: done (2 left) 2024-09-16T04:08:11.222 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 25 current snap is 3 2024-09-16T04:08:11.222 INFO:tasks.rados.rados.0.smithi016.stdout:110: seq_num 73 ranges {1359872=458752,1818624=475136,2293760=294912} 2024-09-16T04:08:11.228 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:11.232 INFO:tasks.rados.rados.0.smithi016.stdout:110: writing smithi01639990-25 from 1359872 to 1818624 tid 1 2024-09-16T04:08:11.235 INFO:tasks.rados.rados.0.smithi016.stdout:110: writing smithi01639990-25 from 1818624 to 2293760 tid 2 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:110: writing smithi01639990-25 from 2293760 to 2588672 tid 3 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:104: finishing write tid 4 to smithi01639990-11 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:104: finishing write tid 5 to smithi01639990-11 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:104: oid 11 updating version 0 to 41 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:104: oid 11 version 41 is already newer than 38 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:104: oid 11 version 41 is already newer than 39 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:104: oid 11 version 41 is already newer than 40 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 11 v 41 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:104: left oid 11 (ObjNum 71 snap 3 seq_num 71) 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:104: done (2 left) 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:111: rmattr oid 47 current snap is 3 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:111: done (2 left) 2024-09-16T04:08:11.238 INFO:tasks.rados.rados.0.smithi016.stdout:112: delete oid 10 current snap is 3 2024-09-16T04:08:11.240 INFO:tasks.rados.rados.0.smithi016.stdout:100: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:08:11.257 INFO:tasks.rados.rados.0.smithi016.stdout:110: finishing write tid 1 to smithi01639990-25 2024-09-16T04:08:11.258 INFO:tasks.rados.rados.0.smithi016.stdout:110: finishing write tid 2 to smithi01639990-25 2024-09-16T04:08:11.317 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:11.316+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:11.372 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:11.372 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:11.381 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:11.414 INFO:tasks.rados.rados.0.smithi016.stdout:110: finishing write tid 3 to smithi01639990-25 2024-09-16T04:08:11.415 INFO:tasks.rados.rados.0.smithi016.stdout:110: finishing write tid 4 to smithi01639990-25 2024-09-16T04:08:11.418 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:11.416+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:11.441 INFO:teuthology.orchestra.run.smithi016.stdout:moved 0 / 146 (0%) 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout:avg 12.1667 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 2.33928 -> 2.33928 (expected baseline 3.33958) 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.2 with 7 -> 7 pgs (0.575342 -> 0.575342 * mean) 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.1 with 16 -> 16 pgs (1.31507 -> 1.31507 * mean) 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout:oload 1000 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 3 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0004 2024-09-16T04:08:11.442 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0035 2024-09-16T04:08:11.443 INFO:teuthology.orchestra.run.smithi016.stderr:no change 2024-09-16T04:08:11.454 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T04:08:11.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-09-16T04:08:11.519 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:11.518+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:11.524 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:11.524 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:11.533 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:11.620 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:11.618+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:11.676 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:11.676 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:11.685 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:11.728 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:11.727+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:11.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:11.789 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:11.789 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:11.797 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:11.803 INFO:tasks.rados.rados.0.smithi016.stdout:110: finishing write tid 5 to smithi01639990-25 2024-09-16T04:08:11.804 INFO:tasks.rados.rados.0.smithi016.stdout:110: oid 25 updating version 0 to 22 2024-09-16T04:08:11.804 INFO:tasks.rados.rados.0.smithi016.stdout:110: oid 25 version 22 is already newer than 20 2024-09-16T04:08:11.804 INFO:tasks.rados.rados.0.smithi016.stdout:110: oid 25 version 22 is already newer than 19 2024-09-16T04:08:11.804 INFO:tasks.rados.rados.0.smithi016.stdout:110: oid 25 version 22 is already newer than 21 2024-09-16T04:08:11.804 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 25 v 22 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-09-16T04:08:11.804 INFO:tasks.rados.rados.0.smithi016.stdout:110: left oid 25 (ObjNum 73 snap 3 seq_num 73) 2024-09-16T04:08:11.819 INFO:tasks.rados.rados.0.smithi016.stdout:100: done (2 left) 2024-09-16T04:08:11.820 INFO:tasks.rados.rados.0.smithi016.stdout:110: done (1 left) 2024-09-16T04:08:11.820 INFO:tasks.rados.rados.0.smithi016.stdout:112: done (0 left) 2024-09-16T04:08:11.820 INFO:tasks.rados.rados.0.smithi016.stdout:113: snap_create 2024-09-16T04:08:11.822 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:11.820+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:11.876 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:11.897 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:11.898 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:11.918 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:11.919 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:11.924 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:11.922+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:11.938 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:11.939 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:11.941 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:11.942 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:11.951 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:11.960 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:11.961 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:11.982 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:12.003 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:12.003 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:12.024 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:12.022+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:12.024 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:12.025 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:12.046 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:12.054 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:12.054 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:12.054 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:12.054 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:12.054 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:12.062 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:12.068 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:12.068 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:12.090 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:12.090 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-16T04:08:12.101 INFO:tasks.rados.rados.0.smithi016.stdout:113: done (0 left) 2024-09-16T04:08:12.101 INFO:tasks.rados.rados.0.smithi016.stdout:114: snap_remove snap 2 2024-09-16T04:08:12.102 INFO:teuthology.orchestra.run.smithi016.stdout:moved 0 / 146 (0%) 2024-09-16T04:08:12.102 INFO:teuthology.orchestra.run.smithi016.stdout:avg 12.1667 2024-09-16T04:08:12.102 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 2.33928 -> 2.33928 (expected baseline 3.33958) 2024-09-16T04:08:12.102 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.2 with 7 -> 7 pgs (0.575342 -> 0.575342 * mean) 2024-09-16T04:08:12.102 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.1 with 16 -> 16 pgs (1.31507 -> 1.31507 * mean) 2024-09-16T04:08:12.102 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:08:12.102 INFO:teuthology.orchestra.run.smithi016.stdout:oload 110 2024-09-16T04:08:12.102 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 0.05 2024-09-16T04:08:12.103 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-16T04:08:12.103 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0004 2024-09-16T04:08:12.103 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0004 2024-09-16T04:08:12.103 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-16T04:08:12.114 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:12.116 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T04:08:12.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-09-16T04:08:12.125 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:12.123+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:12.169 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpRead(3.ds3 32/31 ECSubRead(tid=149, to_read={3:bc502819:::smithi01639990-5:head=0,262144,0}, subchunks={3:bc502819:::smithi01639990-5:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.106913+0000", 2024-09-16T04:08:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.0601025320000002, 2024-09-16T04:08:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057235299999999995, 2024-09-16T04:08:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.106913+0000", 2024-09-16T04:08:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.106913+0000", 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.106914+0000", 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3680000000000001e-06 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.106929+0000", 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4618e-05 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.106981+0000", 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2679000000000002e-05 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.106993+0000", 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1688000000000001e-05 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.107079+0000", 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6360999999999995e-05 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.107485+0000", 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00040563900000000001 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpRead(3.ds3 32/31 ECSubRead(tid=150, to_read={3:bc8571f8:::smithi01639990-35:head=0,262144,0}, subchunks={3:bc8571f8:::smithi01639990-35:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.679910+0000", 2024-09-16T04:08:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.4871054639999999, 2024-09-16T04:08:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00097394799999999998, 2024-09-16T04:08:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679910+0000", 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679910+0000", 2024-09-16T04:08:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679910+0000", 2024-09-16T04:08:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.47e-07 2024-09-16T04:08:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679919+0000", 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2689999999999999e-06 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679971+0000", 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1968000000000001e-05 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679981+0000", 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0453e-05 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.680061+0000", 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9979000000000001e-05 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.680884+0000", 2024-09-16T04:08:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000822432 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 32/27 ECSubWrite(tid=34, reqid=client.4440.0:182, at_version=32'8, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.184503+0000", 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9825116129999998, 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00144147, 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.184503+0000", 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.184503+0000", 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.184505+0000", 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.057e-06 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.184578+0000", 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3386999999999994e-05 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.184625+0000", 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6851999999999998e-05 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.184633+0000", 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7570000000000004e-06 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.184697+0000", 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4331999999999993e-05 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.184709+0000", 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1929999999999999e-05 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185914+0000", 2024-09-16T04:08:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012057039999999999 2024-09-16T04:08:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185945+0000", 2024-09-16T04:08:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0451e-05 2024-09-16T04:08:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 32/27 ECSubWrite(tid=35, reqid=client.4440.0:183, at_version=32'9, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.186313+0000", 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9807022070000002, 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014705639999999999, 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.186313+0000", 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.186313+0000", 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.186314+0000", 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0299999999999998e-07 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.186400+0000", 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5953999999999997e-05 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.186438+0000", 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8402999999999998e-05 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.186447+0000", 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8319999999999995e-06 2024-09-16T04:08:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.186515+0000", 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7848000000000005e-05 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.186525+0000", 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0319999999999999e-05 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187749+0000", 2024-09-16T04:08:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012239600000000001 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187783+0000", 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4443999999999998e-05 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 32/27 ECSubWrite(tid=36, reqid=client.4440.0:184, at_version=32'10, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.187423+0000", 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.979591959, 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00074869200000000002, 2024-09-16T04:08:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187423+0000", 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187423+0000", 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187424+0000", 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.92e-07 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187441+0000", 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7085999999999999e-05 2024-09-16T04:08:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187484+0000", 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3114000000000001e-05 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187492+0000", 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1429999999999998e-06 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187557+0000", 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4633000000000003e-05 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187566+0000", 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7900000000000005e-06 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.188145+0000", 2024-09-16T04:08:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057953200000000001 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.188172+0000", 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6701999999999999e-05 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 32/27 ECSubWrite(tid=37, reqid=client.4440.0:184, at_version=0'0, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.191753+0000", 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9752616180000002, 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057695100000000003, 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191753+0000", 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191753+0000", 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191754+0000", 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7299999999999995e-07 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191762+0000", 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8539999999999997e-06 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191792+0000", 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0341999999999999e-05 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191801+0000", 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2900000000000002e-06 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191859+0000", 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8519e-05 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191868+0000", 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4400000000000005e-06 2024-09-16T04:08:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.192299+0000", 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000431023 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.192330+0000", 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1810000000000002e-05 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:193 3.as0 3:5d02464a:::smithi01639990-40:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9704283780000003, 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041575429999999997, 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 193 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196589+0000", 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8160000000000001e-06 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196600+0000", 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1545e-05 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196613+0000", 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.34e-05 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196621+0000", 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4909999999999999e-06 2024-09-16T04:08:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196682+0000", 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1074000000000005e-05 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196776+0000", 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4102000000000001e-05 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.197563+0000", 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000786446 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.198374+0000", 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081128199999999998 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200668+0000", 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0022936979999999998 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200744+0000", 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6688999999999997e-05 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 32/27 ECSubWriteReply(tid=12, last_complete=32'8, committed=1, applied=1))", 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.200487+0000", 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9665284729999999, 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00058551700000000003, 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200487+0000", 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200487+0000", 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200487+0000", 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1500000000000003e-07 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200492+0000", 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5090000000000004e-06 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200524+0000", 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2042000000000003e-05 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200531+0000", 2024-09-16T04:08:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.199e-06 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200582+0000", 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0355000000000002e-05 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201072+0000", 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049059699999999995 2024-09-16T04:08:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 32/27 ECSubWriteReply(tid=13, last_complete=32'8, committed=1, applied=1))", 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.201702+0000", 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9653129480000002, 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000183203, 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201702+0000", 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201702+0000", 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201703+0000", 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6599999999999996e-07 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201707+0000", 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7689999999999998e-06 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201726+0000", 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9286000000000001e-05 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201734+0000", 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6850000000000001e-06 2024-09-16T04:08:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201870+0000", 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013628100000000001 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201885+0000", 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5515999999999998e-05 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 32/27 ECSubWriteReply(tid=13, last_complete=32'8, committed=1, applied=1))", 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.201709+0000", 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9653059530000001, 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000209509, 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201709+0000", 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201709+0000", 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201710+0000", 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7599999999999997e-07 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201713+0000", 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2040000000000002e-06 2024-09-16T04:08:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201736+0000", 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.34e-05 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201747+0000", 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0199e-05 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201899+0000", 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015242500000000001 2024-09-16T04:08:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.201919+0000", 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9704999999999999e-05 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 32/27 ECSubWrite(tid=11, reqid=client.4440.0:199, at_version=32'8, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.224860+0000", 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9421553949999999, 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014447399999999999, 2024-09-16T04:08:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.224860+0000", 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.224860+0000", 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.224861+0000", 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.432e-06 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.224980+0000", 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011863500000000001 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.225014+0000", 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4499000000000003e-05 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.225023+0000", 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.225103+0000", 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0107999999999999e-05 2024-09-16T04:08:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.225127+0000", 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4173999999999999e-05 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226271+0000", 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001144557 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226304+0000", 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2963999999999998e-05 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 32/27 ECSubWrite(tid=12, reqid=client.4440.0:199, at_version=0'0, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.693801+0000", 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.4732141400000001, 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00078437000000000005, 2024-09-16T04:08:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693801+0000", 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693801+0000", 2024-09-16T04:08:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693802+0000", 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.06e-06 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693823+0000", 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0744e-05 2024-09-16T04:08:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693849+0000", 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6319999999999999e-05 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693859+0000", 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7739999999999999e-06 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693935+0000", 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6415000000000006e-05 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693946+0000", 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1198e-05 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.694556+0000", 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00060992200000000005 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.694585+0000", 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8937000000000001e-05 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 33/27 ECSubWrite(tid=13, reqid=osd.4.0:13, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.095754+0000", 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.0712607080000001, 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013393649999999999, 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095754+0000", 2024-09-16T04:08:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095754+0000", 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095755+0000", 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3500000000000005e-07 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095778+0000", 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3065999999999999e-05 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095820+0000", 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1344000000000003e-05 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095870+0000", 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0318999999999997e-05 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.096054+0000", 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018359800000000001 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.096063+0000", 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7480000000000005e-06 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.097059+0000", 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000995337 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.097094+0000", 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:08:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.0633116989999998, 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0068480019999999997, 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103704+0000", 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7100000000000001e-07 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103727+0000", 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3694999999999999e-05 2024-09-16T04:08:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103810+0000", 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3091999999999997e-05 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103816+0000", 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4519999999999996e-06 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103971+0000", 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000154993 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103982+0000", 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0933e-05 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110491+0000", 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065096800000000003 2024-09-16T04:08:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110551+0000", 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9985999999999998e-05 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.063262361, 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067325809999999996, 2024-09-16T04:08:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103754+0000", 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5899999999999995e-07 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103768+0000", 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4562999999999999e-05 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103805+0000", 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6730000000000002e-05 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103813+0000", 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103868+0000", 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4475000000000003e-05 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103877+0000", 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7099999999999996e-06 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110444+0000", 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065675170000000001 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110485+0000", 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1177000000000002e-05 2024-09-16T04:08:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 33/27 ECSubWrite(tid=41, reqid=osd.5.0:39, at_version=0'0, trim_to=0'0, roll_forward_to=33'12))", 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.117399+0000", 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.0496165039999998, 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001889462, 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117399+0000", 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117399+0000", 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117400+0000", 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1319999999999999e-06 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117414+0000", 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4039999999999999e-05 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117456+0000", 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2691999999999997e-05 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117468+0000", 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1245e-05 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117541+0000", 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3243000000000003e-05 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117551+0000", 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0032e-05 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.119240+0000", 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016889489999999999 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.119288+0000", 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8128999999999999e-05 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/27 ECSubWrite(tid=84, reqid=client.4440.0:215, at_version=33'14, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.0283027420000002, 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042300860000000001, 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5900000000000001e-07 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139362+0000", 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064961800000000003 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139387+0000", 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5369999999999999e-05 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139394+0000", 2024-09-16T04:08:12.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2190000000000001e-06 2024-09-16T04:08:12.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139458+0000", 2024-09-16T04:08:12.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4603000000000005e-05 2024-09-16T04:08:12.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139470+0000", 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1810999999999999e-05 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142910+0000", 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003439896 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142942+0000", 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2410000000000003e-05 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/27 ECSubWrite(tid=85, reqid=client.4440.0:216, at_version=33'15, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:12.254 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.0261004690000002, 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043866510000000001, 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141225+0000", 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00030972200000000001 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141258+0000", 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3228000000000001e-05 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141267+0000", 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5590000000000006e-06 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141327+0000", 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0100999999999999e-05 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141338+0000", 2024-09-16T04:08:12.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0745e-05 2024-09-16T04:08:12.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145260+0000", 2024-09-16T04:08:12.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039227610000000003 2024-09-16T04:08:12.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145301+0000", 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0884999999999999e-05 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.260 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.141651+0000", 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.0253639539999999, 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036743629999999999, 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141651+0000", 2024-09-16T04:08:12.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141651+0000", 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141652+0000", 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2099999999999996e-07 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141669+0000", 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7410000000000001e-05 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141695+0000", 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5982e-05 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141700+0000", 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9010000000000002e-06 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142744+0000", 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010444250000000001 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142758+0000", 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3672000000000001e-05 2024-09-16T04:08:12.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145304+0000", 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00254631 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145325+0000", 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1041999999999999e-05 2024-09-16T04:08:12.265 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 33/27 ECSubWriteReply(tid=14, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.143705+0000", 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.0233101840000001, 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000396191, 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.266 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.143705+0000", 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.143705+0000", 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.143705+0000", 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8899999999999999e-07 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.143709+0000", 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6399999999999999e-06 2024-09-16T04:08:12.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.143735+0000", 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5601000000000001e-05 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.143742+0000", 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0079999999999996e-06 2024-09-16T04:08:12.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.143789+0000", 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7349999999999999e-05 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.144101+0000", 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00031200299999999999 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.270 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.271 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:12.272 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:12.272 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:12.269+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:12.373 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:12.371+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:12.416 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:12.416 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:12.416 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:12.416 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:12.425 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout:moved 0 / 146 (0%) 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout:avg 12.1667 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 2.33928 -> 2.33928 (expected baseline 3.33958) 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.2 with 7 -> 7 pgs (0.575342 -> 0.575342 * mean) 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.1 with 16 -> 16 pgs (1.31507 -> 1.31507 * mean) 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout:oload 110 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 3 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-16T04:08:12.457 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0004 2024-09-16T04:08:12.458 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0004 2024-09-16T04:08:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:no change 2024-09-16T04:08:12.470 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T04:08:12.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2024-09-16T04:08:12.473 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:12.472+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:12.528 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:12.528 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:12.528 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:12.528 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:12.528 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:12.537 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:12.574 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:12.573+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:12.642 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.699043194000001, 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.716638951, 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.706276445, 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:51.800386+0000", 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.839379666999999, 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086667150000000002, 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 59 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800386+0000", 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800386+0000", 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800387+0000", 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9999999999999997e-07 2024-09-16T04:08:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800392+0000", 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.899e-06 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800400+0000", 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5410000000000003e-06 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.800405+0000", 2024-09-16T04:08:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.823e-06 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.801125+0000", 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00071925799999999996 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.801175+0000", 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0037999999999998e-05 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.801263+0000", 2024-09-16T04:08:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8392000000000003e-05 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.802218+0000", 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095508799999999999 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.809001+0000", 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067828769999999997 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.809031+0000", 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9920000000000002e-05 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.809053+0000", 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2178999999999999e-05 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.209688393, 2024-09-16T04:08:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.186175491, 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092583400000000003, 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-09-16T04:08:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.37e-07 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453604+0000", 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2658e-05 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453611+0000", 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0650000000000001e-06 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453616+0000", 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9099999999999996e-06 2024-09-16T04:08:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459323+0000", 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057072879999999996 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459369+0000", 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6436000000000001e-05 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459442+0000", 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2459000000000005e-05 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460087+0000", 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064517999999999997 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462796+0000", 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027087639999999998 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462827+0000", 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0827000000000001e-05 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462849+0000", 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2215999999999999e-05 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.755379302000001, 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088271749999999996, 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 158 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5899999999999996e-07 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884453+0000", 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999997e-05 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884462+0000", 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2920000000000001e-06 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884468+0000", 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9399999999999999e-06 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886225+0000", 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0017572079999999999 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886269+0000", 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3118999999999998e-05 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886334+0000", 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5695e-05 2024-09-16T04:08:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888544+0000", 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002210086 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893152+0000", 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046078940000000004 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893177+0000", 2024-09-16T04:08:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.463e-05 2024-09-16T04:08:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893214+0000", 2024-09-16T04:08:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7002000000000002e-05 2024-09-16T04:08:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.755284273000001, 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087992620000000004, 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-09-16T04:08:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0500000000000003e-07 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884486+0000", 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-06 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884493+0000", 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8759999999999999e-06 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884498+0000", 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3550000000000004e-06 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888576+0000", 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040780230000000001 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888629+0000", 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2892000000000001e-05 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888704+0000", 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5024000000000005e-05 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892724+0000", 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040194619999999997 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893222+0000", 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049801700000000001 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893246+0000", 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3901e-05 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893281+0000", 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5277000000000001e-05 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.755256238000001, 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088385849999999995, 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 160 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.698 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3900000000000005e-07 2024-09-16T04:08:12.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884514+0000", 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2169999999999999e-06 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884520+0000", 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8639999999999998e-06 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884527+0000", 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.866e-06 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888915+0000", 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043874270000000002 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888953+0000", 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7839000000000001e-05 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889014+0000", 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1289000000000002e-05 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892771+0000", 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037575199999999999 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893289+0000", 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00051718599999999999 2024-09-16T04:08:12.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893314+0000", 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5202999999999999e-05 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893348+0000", 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4635000000000003e-05 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.703 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.755219058000002, 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088608530000000001, 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 161 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884548+0000", 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5599999999999995e-07 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884603+0000", 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5615999999999997e-05 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884612+0000", 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6589999999999996e-06 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884619+0000", 2024-09-16T04:08:12.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4630000000000004e-06 2024-09-16T04:08:12.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889203+0000", 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045841989999999997 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889242+0000", 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8713e-05 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889304+0000", 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1858000000000003e-05 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892819+0000", 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003515107 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893354+0000", 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053514000000000003 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893377+0000", 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2265000000000001e-05 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893408+0000", 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1276999999999999e-05 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.710 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.755139808999999, 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094233230000000008, 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 162 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884627+0000", 2024-09-16T04:08:12.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884629+0000", 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.187e-06 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884632+0000", 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.501e-06 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884635+0000", 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7389999999999999e-06 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892223+0000", 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075881029999999997 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892291+0000", 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7508000000000005e-05 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892412+0000", 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012124 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893979+0000", 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015676069999999999 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893987+0000", 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.306e-06 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894013+0000", 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6305999999999999e-05 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894050+0000", 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6492999999999998e-05 2024-09-16T04:08:12.716 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.755127437999999, 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095013200000000006, 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.717 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 163 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:12.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2100000000000001e-07 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884642+0000", 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.478e-06 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884646+0000", 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.292e-06 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884648+0000", 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6230000000000001e-06 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892847+0000", 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0081990580000000004 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892891+0000", 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3628999999999999e-05 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892959+0000", 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8151000000000005e-05 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894055+0000", 2024-09-16T04:08:12.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001095927 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894088+0000", 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3016999999999999e-05 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894117+0000", 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9204999999999999e-05 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894140+0000", 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2719000000000001e-05 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.755113900000001, 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.723 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:12.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:12.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.755059839000001, 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.729 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:12.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.5307913009999998, 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:12.735 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.9528573409999996, 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:12.739 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:12.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.743 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.744 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.744 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s2 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:12.744 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:12.744 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.540083112, 2024-09-16T04:08:12.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010866819, 2024-09-16T04:08:12.744 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.744 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.745 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.745 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:12.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099684+0000", 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.004e-06 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099714+0000", 2024-09-16T04:08:12.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9807000000000001e-05 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099769+0000", 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5306000000000002e-05 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099781+0000", 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2091e-05 2024-09-16T04:08:12.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099835+0000", 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4058000000000003e-05 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099845+0000", 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5729999999999997e-06 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110509+0000", 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010664339 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110550+0000", 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640999999999999e-05 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.749 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.5360441959999998, 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.750 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:12.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.5346567819999999, 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:12.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:12.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:12.759 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.532389727, 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099172860000000009, 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107377+0000", 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8900000000000001e-07 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:12.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107437+0000", 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0628000000000003e-05 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107526+0000", 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8873000000000001e-05 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107535+0000", 2024-09-16T04:08:12.762 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0489999999999993e-06 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111543+0000", 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040080810000000001 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111554+0000", 2024-09-16T04:08:12.763 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0321e-05 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117271+0000", 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057176529999999996 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117294+0000", 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2391999999999998e-05 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.764 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.765 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:12.765 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:12.765 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:12.766 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:12.767 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:12.764+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:12.803 INFO:teuthology.orchestra.run.smithi016.stdout:moved 0 / 146 (0%) 2024-09-16T04:08:12.803 INFO:teuthology.orchestra.run.smithi016.stdout:avg 12.1667 2024-09-16T04:08:12.803 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 2.33928 -> 2.33928 (expected baseline 3.33958) 2024-09-16T04:08:12.804 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.2 with 7 -> 7 pgs (0.575342 -> 0.575342 * mean) 2024-09-16T04:08:12.804 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.1 with 16 -> 16 pgs (1.31507 -> 1.31507 * mean) 2024-09-16T04:08:12.804 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:08:12.804 INFO:teuthology.orchestra.run.smithi016.stdout:oload 110 2024-09-16T04:08:12.804 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 1 2024-09-16T04:08:12.804 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-16T04:08:12.804 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0004 2024-09-16T04:08:12.804 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0004 2024-09-16T04:08:12.804 INFO:teuthology.orchestra.run.smithi016.stderr:no change 2024-09-16T04:08:12.816 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-16T04:08:12.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-09-16T04:08:12.867 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:12.866+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:12.882 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:12.882 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:12.882 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:12.882 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:12.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:12.969 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:12.967+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:13.035 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:13.035 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:13.035 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:13.035 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:13.035 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:13.044 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:13.087 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:13.085+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:13.103 INFO:tasks.rados.rados.0.smithi016.stdout:114: done (0 left) 2024-09-16T04:08:13.103 INFO:tasks.rados.rados.0.smithi016.stdout:115: rollback oid 36 current snap is 4 2024-09-16T04:08:13.103 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 36 to 3 2024-09-16T04:08:13.103 INFO:tasks.rados.rados.0.smithi016.stdout:116: snap_create 2024-09-16T04:08:13.108 INFO:tasks.rados.rados.0.smithi016.stdout:115: finishing rollback tid 1 to smithi01639990-36 2024-09-16T04:08:13.108 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 36 v 17 (ObjNum 129 snap 0 seq_num 3716826996) dirty dne 2024-09-16T04:08:13.117 INFO:teuthology.orchestra.run.smithi016.stdout:moved 0 / 146 (0%) 2024-09-16T04:08:13.118 INFO:teuthology.orchestra.run.smithi016.stdout:avg 12.1667 2024-09-16T04:08:13.118 INFO:teuthology.orchestra.run.smithi016.stdout:stddev 2.33928 -> 2.33928 (expected baseline 3.33958) 2024-09-16T04:08:13.118 INFO:teuthology.orchestra.run.smithi016.stdout:min osd.2 with 7 -> 7 pgs (0.575342 -> 0.575342 * mean) 2024-09-16T04:08:13.118 INFO:teuthology.orchestra.run.smithi016.stdout:max osd.1 with 16 -> 16 pgs (1.31507 -> 1.31507 * mean) 2024-09-16T04:08:13.118 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:08:13.118 INFO:teuthology.orchestra.run.smithi016.stdout:oload 1000 2024-09-16T04:08:13.119 INFO:teuthology.orchestra.run.smithi016.stdout:max_change 3 2024-09-16T04:08:13.119 INFO:teuthology.orchestra.run.smithi016.stdout:max_change_osds 4 2024-09-16T04:08:13.119 INFO:teuthology.orchestra.run.smithi016.stdout:average_utilization 0.0004 2024-09-16T04:08:13.119 INFO:teuthology.orchestra.run.smithi016.stdout:overload_utilization 0.0035 2024-09-16T04:08:13.120 INFO:teuthology.orchestra.run.smithi016.stderr:no change 2024-09-16T04:08:13.157 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:13.157 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:13.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:13.157 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:13.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.157 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:186 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~589824] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:13.157 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.194343+0000", 2024-09-16T04:08:13.157 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9600484800000002, 2024-09-16T04:08:13.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060370950000000001, 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 186 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.194343+0000", 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.194343+0000", 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.194345+0000", 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.759e-06 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195173+0000", 2024-09-16T04:08:13.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00082836499999999996 2024-09-16T04:08:13.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195190+0000", 2024-09-16T04:08:13.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6756999999999999e-05 2024-09-16T04:08:13.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195198+0000", 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2599999999999999e-06 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195250+0000", 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1996999999999997e-05 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195323+0000", 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3161e-05 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196249+0000", 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00092679299999999997 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200145+0000", 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038957269999999999 2024-09-16T04:08:13.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200279+0000", 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013368599999999999 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200380+0000", 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010158999999999999 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:187 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~376832] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.196468+0000", 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9579237819999999, 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053497429999999997, 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 187 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.164 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196468+0000", 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196468+0000", 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196469+0000", 2024-09-16T04:08:13.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.299e-06 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197470+0000", 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010008770000000001 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197486+0000", 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6186000000000001e-05 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197496+0000", 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.268e-06 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197604+0000", 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108723 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:13.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197700+0000", 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5773999999999995e-05 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198516+0000", 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081628899999999999 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200458+0000", 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001941826 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201745+0000", 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001287049 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201818+0000", 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2452000000000004e-05 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:188 3.9s0 3:92dcccbe:::smithi01639990-16:head [setxattr _header (56) in=7b] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:13.169 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.197513+0000", 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9568784109999999, 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044478620000000003, 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 188 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197513+0000", 2024-09-16T04:08:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197513+0000", 2024-09-16T04:08:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197514+0000", 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8599999999999996e-07 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197518+0000", 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.817e-06 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197526+0000", 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4449999999999999e-06 2024-09-16T04:08:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197531+0000", 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6579999999999999e-06 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199161+0000", 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001630086 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:13.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199239+0000", 2024-09-16T04:08:13.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7812000000000006e-05 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.199687+0000", 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00044821500000000002 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200889+0000", 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001201616 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201903+0000", 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010135459999999999 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201961+0000", 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8681000000000003e-05 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.176 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:189 3.9s0 3:92dcccbe:::smithi01639990-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:13.177 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.197539+0000", 2024-09-16T04:08:13.177 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9568527309999997, 2024-09-16T04:08:13.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005956355, 2024-09-16T04:08:13.177 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.177 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.177 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:13.177 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:13.177 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:13.177 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 189 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197539+0000", 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197539+0000", 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197541+0000", 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0770000000000002e-06 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197543+0000", 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1859999999999999e-06 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197550+0000", 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7020000000000004e-06 2024-09-16T04:08:13.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197555+0000", 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7809999999999996e-06 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200575+0000", 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030196649999999999 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200666+0000", 2024-09-16T04:08:13.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1508999999999994e-05 2024-09-16T04:08:13.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202294+0000", 2024-09-16T04:08:13.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016276719999999999 2024-09-16T04:08:13.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:13.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202368+0000", 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4441000000000001e-05 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.203495+0000", 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001127322 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(osd.11.18:1 3.9s0 3:92dcccbe:::smithi01639990-16:head [assert-version v23,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:13.182 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.214792+0000", 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9395998150000002, 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063663399999999998, 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214792+0000", 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214792+0000", 2024-09-16T04:08:13.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214794+0000", 2024-09-16T04:08:13.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9640000000000002e-06 2024-09-16T04:08:13.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214802+0000", 2024-09-16T04:08:13.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9699999999999999e-06 2024-09-16T04:08:13.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214822+0000", 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9737000000000001e-05 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214828+0000", 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0159999999999997e-06 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214901+0000", 2024-09-16T04:08:13.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3320999999999998e-05 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214983+0000", 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2216e-05 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.221158+0000", 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061751159999999996 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 32/27 ECSubWrite(tid=27, reqid=client.4440.0:196, at_version=0'0, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.214858+0000", 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.939533366, 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025191089999999998, 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214858+0000", 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214858+0000", 2024-09-16T04:08:13.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214859+0000", 2024-09-16T04:08:13.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0999999999999997e-07 2024-09-16T04:08:13.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214869+0000", 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4020000000000008e-06 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214896+0000", 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6940999999999998e-05 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214904+0000", 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3999999999999992e-06 2024-09-16T04:08:13.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214958+0000", 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4146e-05 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214967+0000", 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3809999999999999e-06 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217348+0000", 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023814420000000001 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217378+0000", 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9587e-05 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.9s0 32/27 ECSubRead(tid=36, to_read={3:92dcccbe:::smithi01639990-16:head=0,520192,0}, subchunks={3:92dcccbe:::smithi01639990-16:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9391876899999998, 2024-09-16T04:08:13.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053324849999999997, 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:13.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215226+0000", 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1671e-05 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215235+0000", 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.329e-06 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215280+0000", 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5154000000000002e-05 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220537+0000", 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0052563310000000004 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpReadReply(3.9s0 32/27 ECSubReadReply(tid=36, attrs_read=0))", 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.216199+0000", 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9381926869999999, 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044083860000000002, 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.199 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216199+0000", 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216199+0000", 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216200+0000", 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3900000000000004e-07 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217297+0000", 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010969560000000001 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217324+0000", 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7365999999999999e-05 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217330+0000", 2024-09-16T04:08:13.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.631e-06 2024-09-16T04:08:13.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220576+0000", 2024-09-16T04:08:13.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032459300000000002 2024-09-16T04:08:13.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220608+0000", 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1664e-05 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpReadReply(3.9s0 32/27 ECSubReadReply(tid=36, attrs_read=0))", 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.216528+0000", 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9378633599999997, 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041487570000000003, 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.203 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216528+0000", 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216528+0000", 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216529+0000", 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6599999999999996e-07 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217353+0000", 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00082443000000000002 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217380+0000", 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6608e-05 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.217386+0000", 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.665e-06 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220651+0000", 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032651540000000001 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220677+0000", 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6333999999999999e-05 2024-09-16T04:08:13.206 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpReadReply(3.9s0 32/27 ECSubReadReply(tid=36, attrs_read=0))", 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.218199+0000", 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9361923550000002, 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0025318659999999998, 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.218199+0000", 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.218199+0000", 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.218200+0000", 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9300000000000006e-07 2024-09-16T04:08:13.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.218599+0000", 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039837899999999997 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.218627+0000", 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8445e-05 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.218633+0000", 2024-09-16T04:08:13.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.083e-06 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220706+0000", 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002072646 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220731+0000", 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5320000000000002e-05 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.211 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es1 32/27 ECSubWrite(tid=28, reqid=client.4440.0:197, at_version=32'4, trim_to=0'0, roll_forward_to=30'2))", 2024-09-16T04:08:13.212 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.227996+0000", 2024-09-16T04:08:13.212 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9263955770000001, 2024-09-16T04:08:13.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045941999999999997, 2024-09-16T04:08:13.212 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.212 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.212 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227996+0000", 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227996+0000", 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227998+0000", 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9719999999999999e-06 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228266+0000", 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00026775200000000001 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228320+0000", 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4098000000000003e-05 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228341+0000", 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1206e-05 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228422+0000", 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0264000000000003e-05 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228440+0000", 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8111999999999999e-05 2024-09-16T04:08:13.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232550+0000", 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041099320000000002 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232590+0000", 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0864e-05 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es1 33/27 ECSubWrite(tid=31, reqid=osd.11.0:31, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.102012+0000", 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.0523794400000002, 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035127769999999999, 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102012+0000", 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102012+0000", 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102013+0000", 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4499999999999999e-07 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102041+0000", 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8588000000000001e-05 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102396+0000", 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00035513800000000001 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102403+0000", 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2500000000000003e-06 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102450+0000", 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7555000000000001e-05 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102459+0000", 2024-09-16T04:08:13.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3049999999999998e-06 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105490+0000", 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003031064 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105525+0000", 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4731999999999999e-05 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s4 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.102055+0000", 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.0523364239999999, 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044053670000000003, 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102055+0000", 2024-09-16T04:08:13.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102055+0000", 2024-09-16T04:08:13.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102056+0000", 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6299999999999999e-07 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102070+0000", 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4321000000000001e-05 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102920+0000", 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000850409 2024-09-16T04:08:13.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102928+0000", 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2189999999999998e-06 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102985+0000", 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6978000000000002e-05 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102993+0000", 2024-09-16T04:08:13.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9950000000000005e-06 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106426+0000", 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034338799999999998 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106461+0000", 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4402e-05 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103462+0000", 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.0509303299999999, 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047755310000000004, 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103462+0000", 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103462+0000", 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103462+0000", 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7899999999999997e-07 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103488+0000", 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5867000000000002e-05 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103564+0000", 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5828000000000007e-05 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103571+0000", 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3520000000000001e-06 2024-09-16T04:08:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.104062+0000", 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00049022900000000001 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.104071+0000", 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8560000000000002e-06 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108206+0000", 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041343020000000003 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108237+0000", 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1318000000000001e-05 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:13.231 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.0508983719999998, 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073296819999999997, 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:13.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103494+0000", 2024-09-16T04:08:13.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1399999999999997e-07 2024-09-16T04:08:13.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103514+0000", 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9698999999999999e-05 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103592+0000", 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8378000000000001e-05 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103598+0000", 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9100000000000002e-06 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105790+0000", 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021916359999999998 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105798+0000", 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2490000000000001e-06 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110788+0000", 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049902660000000001 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110823+0000", 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5129999999999997e-05 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es1 33/27 ECSubWrite(tid=33, reqid=osd.11.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=33'6))", 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.114154+0000", 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.0402382760000002, 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027471729999999999, 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.236 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.114154+0000", 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.114154+0000", 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.114154+0000", 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0900000000000001e-07 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.114166+0000", 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1790999999999999e-05 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115790+0000", 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016238909999999999 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115798+0000", 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5979999999999999e-06 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115861+0000", 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3567999999999999e-05 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115879+0000", 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8363999999999999e-05 2024-09-16T04:08:13.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116873+0000", 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099385800000000002 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116901+0000", 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7393999999999999e-05 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=34, reqid=client.4440.0:211, at_version=33'20, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.0386814360000001, 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078650630000000003, 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115711+0000", 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2399999999999998e-07 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115771+0000", 2024-09-16T04:08:13.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9478999999999998e-05 2024-09-16T04:08:13.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119749+0000", 2024-09-16T04:08:13.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039784099999999999 2024-09-16T04:08:13.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119760+0000", 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0664e-05 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119899+0000", 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013905300000000001 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119919+0000", 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9854999999999999e-05 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123544+0000", 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036256499999999998 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123575+0000", 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1328000000000002e-05 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.245 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=35, reqid=client.4440.0:212, at_version=33'21, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.0359263140000001, 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077855709999999998, 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118551+0000", 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4878999999999998e-05 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120595+0000", 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020432639999999999 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120604+0000", 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0250000000000008e-06 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122166+0000", 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015625599999999999 2024-09-16T04:08:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122175+0000", 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8470000000000002e-06 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126217+0000", 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040415249999999998 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126251+0000", 2024-09-16T04:08:13.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4570999999999998e-05 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=84, reqid=client.4440.0:215, at_version=33'14, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.138792+0000", 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.015599903, 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045910380000000004, 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.138792+0000", 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.138792+0000", 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.138792+0000", 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.15e-07 2024-09-16T04:08:13.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.139658+0000", 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00086640700000000003 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141187+0000", 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001528352 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141195+0000", 2024-09-16T04:08:13.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6670000000000002e-06 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141254+0000", 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8572000000000001e-05 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141260+0000", 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3799999999999999e-06 2024-09-16T04:08:13.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.143351+0000", 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020901019999999999 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.143383+0000", 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2443000000000001e-05 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.254 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.0126985509999997, 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:13.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:13.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:13.259 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.259 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.259 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.259 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.259 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.259 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:13.261 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:13.258+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:13.261 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:13.361 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:13.360+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:13.405 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:13.405 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:13.405 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:13.405 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:13.414 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:13.463 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:13.461+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:13.516 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:13.516 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:13.517 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:13.517 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:13.517 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:13.525 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:13.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:13.562+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:13.629 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.5329742700000004, 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.630 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.634 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es5 32/27 ECSubWrite(tid=28, reqid=client.4440.0:197, at_version=32'4, trim_to=0'0, roll_forward_to=30'2))", 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.227608+0000", 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.3996737120000002, 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004862318, 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227608+0000", 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227608+0000", 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227610+0000", 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8369999999999999e-06 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228034+0000", 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042372999999999998 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228094+0000", 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0418000000000003e-05 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228107+0000", 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3121e-05 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228200+0000", 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2559000000000006e-05 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228213+0000", 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3574000000000001e-05 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232430+0000", 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004217213 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232470+0000", 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9866000000000001e-05 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.639 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as5 33/27 ECSubWrite(tid=14, reqid=osd.6.0:14, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.095198+0000", 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.5320833079999998, 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044375830000000002, 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095198+0000", 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095198+0000", 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095199+0000", 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2999999999999999e-07 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095220+0000", 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1407999999999999e-05 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095852+0000", 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063245300000000005 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095862+0000", 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0550000000000005e-06 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096043+0000", 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018101900000000001 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096052+0000", 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7909999999999999e-06 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099602+0000", 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035497279999999998 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099636+0000", 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3899000000000002e-05 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 33/27 ECSubWrite(tid=77, reqid=osd.3.0:77, at_version=33'19, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.097917+0000", 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.5293642680000001, 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038417049999999999, 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.644 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.097917+0000", 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.097917+0000", 2024-09-16T04:08:13.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.097918+0000", 2024-09-16T04:08:13.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2399999999999999e-07 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.097939+0000", 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1843000000000001e-05 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.097974+0000", 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.097984+0000", 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3279999999999995e-06 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098047+0000", 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3350999999999997e-05 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098058+0000", 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0421000000000001e-05 2024-09-16T04:08:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101725+0000", 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036675929999999998 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101759+0000", 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3927000000000002e-05 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.649 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 33/27 ECSubWrite(tid=37, reqid=osd.8.0:37, at_version=33'25, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.098805+0000", 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.5284762900000004, 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043258790000000004, 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098805+0000", 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098805+0000", 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098806+0000", 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.23e-07 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098827+0000", 2024-09-16T04:08:13.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1308e-05 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098879+0000", 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1863000000000001e-05 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098886+0000", 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6669999999999996e-06 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098947+0000", 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0789999999999999e-05 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098956+0000", 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.49e-06 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103100+0000", 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041444289999999998 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103131+0000", 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1109000000000003e-05 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.5275337970000002, 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067064200000000003, 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.654 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099749+0000", 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1299999999999999e-07 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099771+0000", 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2642000000000001e-05 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099811+0000", 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9406999999999997e-05 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099819+0000", 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100401+0000", 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058156199999999999 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100411+0000", 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7079999999999992e-06 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106408+0000", 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059972369999999999 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106454+0000", 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6384000000000002e-05 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es5 33/27 ECSubWrite(tid=31, reqid=osd.11.0:31, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.101895+0000", 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.525386879, 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046241909999999997, 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101895+0000", 2024-09-16T04:08:13.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101895+0000", 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101895+0000", 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.73e-07 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101941+0000", 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5605999999999997e-05 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102038+0000", 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7359000000000001e-05 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102048+0000", 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5740000000000002e-06 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102115+0000", 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7688000000000007e-05 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102125+0000", 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4569999999999994e-06 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106479+0000", 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043546799999999997 2024-09-16T04:08:13.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106519+0000", 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9653999999999997e-05 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.102002+0000", 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.5252800620000002, 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045426470000000004, 2024-09-16T04:08:13.663 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102002+0000", 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102002+0000", 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102002+0000", 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9299999999999999e-07 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102018+0000", 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6099999999999998e-05 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102101+0000", 2024-09-16T04:08:13.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2735e-05 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102108+0000", 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7230000000000004e-06 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102214+0000", 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105642 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102223+0000", 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.149e-06 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106523+0000", 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042998619999999998 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106544+0000", 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1243e-05 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.667 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103167+0000", 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.5241147310000001, 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0055620829999999998, 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103167+0000", 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103167+0000", 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103175+0000", 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2879999999999993e-06 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103717+0000", 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00054184999999999997 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103749+0000", 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2314999999999998e-05 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103757+0000", 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2729999999999997e-06 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103808+0000", 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1072999999999997e-05 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103817+0000", 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8410000000000006e-06 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108704+0000", 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048869250000000003 2024-09-16T04:08:13.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108729+0000", 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5517999999999999e-05 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103841+0000", 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.5234406099999998, 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048534260000000001, 2024-09-16T04:08:13.672 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103841+0000", 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103841+0000", 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103842+0000", 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4700000000000001e-07 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103856+0000", 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4748e-05 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103886+0000", 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9437999999999998e-05 2024-09-16T04:08:13.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103894+0000", 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9189999999999999e-06 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103978+0000", 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3775000000000003e-05 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103983+0000", 2024-09-16T04:08:13.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.519e-06 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108661+0000", 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004677858 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108694+0000", 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3522000000000002e-05 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.107048+0000", 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.5202332629999997, 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040110600000000003, 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107048+0000", 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107048+0000", 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107049+0000", 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8500000000000001e-07 2024-09-16T04:08:13.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107285+0000", 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00023541999999999999 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107306+0000", 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0978000000000001e-05 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107312+0000", 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8240000000000002e-06 2024-09-16T04:08:13.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107349+0000", 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6207000000000001e-05 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107354+0000", 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2789999999999998e-06 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111030+0000", 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003675939 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111059+0000", 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9728000000000001e-05 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141719+0000", 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.4855625589999999, 2024-09-16T04:08:13.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041391529999999996, 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141719+0000", 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141719+0000", 2024-09-16T04:08:13.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141720+0000", 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8000000000000005e-07 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141736+0000", 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6583999999999999e-05 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141767+0000", 2024-09-16T04:08:13.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0485e-05 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141776+0000", 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7539999999999995e-06 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141845+0000", 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9376999999999997e-05 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141855+0000", 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7119999999999995e-06 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.145826+0000", 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039709020000000001 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.145858+0000", 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2558999999999998e-05 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 37/27 ECSubWrite(tid=70, reqid=osd.5.0:70, at_version=37'20, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.102718+0000", 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.52456323299999996, 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040052159999999998, 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102718+0000", 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102718+0000", 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102719+0000", 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.67e-07 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102737+0000", 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8793000000000001e-05 2024-09-16T04:08:13.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102783+0000", 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5433999999999999e-05 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102795+0000", 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2543e-05 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102880+0000", 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4486000000000004e-05 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102892+0000", 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.189e-05 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106685+0000", 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003793177 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106724+0000", 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8726000000000002e-05 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.523086885, 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062253660000000004, 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104196+0000", 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:08:13.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104218+0000", 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1974999999999999e-05 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104245+0000", 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7489999999999999e-05 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104254+0000", 2024-09-16T04:08:13.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104313+0000", 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9151000000000003e-05 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104322+0000", 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:08:13.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110389+0000", 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006067205 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110420+0000", 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1100999999999999e-05 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.106795+0000", 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.52048620899999998, 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056249589999999997, 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106795+0000", 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106795+0000", 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106796+0000", 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.79e-07 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106819+0000", 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3085999999999999e-05 2024-09-16T04:08:13.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106897+0000", 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8067999999999999e-05 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106905+0000", 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2209999999999997e-06 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106978+0000", 2024-09-16T04:08:13.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2822000000000002e-05 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106988+0000", 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0468999999999999e-05 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112382+0000", 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005393424 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112420+0000", 2024-09-16T04:08:13.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8689999999999997e-05 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=163, reqid=osd.7.0:163, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.51730759900000001, 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053772749999999999, 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1799999999999999e-07 2024-09-16T04:08:13.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110009+0000", 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4467000000000001e-05 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110085+0000", 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6359999999999994e-05 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110093+0000", 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0879999999999995e-06 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110482+0000", 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00038852100000000002 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110498+0000", 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5858999999999998e-05 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115317+0000", 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048192579999999999 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115351+0000", 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4504e-05 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.703 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.51727036299999996, 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062070839999999999, 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110012+0000", 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3999999999999997e-07 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110025+0000", 2024-09-16T04:08:13.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3053e-05 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110784+0000", 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075944100000000002 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110793+0000", 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4880000000000007e-06 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110853+0000", 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0333e-05 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110861+0000", 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2090000000000005e-06 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116184+0000", 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053227389999999999 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116218+0000", 2024-09-16T04:08:13.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4480999999999997e-05 2024-09-16T04:08:13.708 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.708 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.708 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.708 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 37/27 ECSubWrite(tid=128, reqid=osd.3.0:124, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:13.709 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111261+0000", 2024-09-16T04:08:13.709 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.51602039899999996, 2024-09-16T04:08:13.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041541820000000002, 2024-09-16T04:08:13.709 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.709 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.709 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.709 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111261+0000", 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111261+0000", 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111262+0000", 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8100000000000002e-07 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111273+0000", 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0592e-05 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111299+0000", 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6585999999999999e-05 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111308+0000", 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7350000000000002e-06 2024-09-16T04:08:13.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111384+0000", 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5816999999999996e-05 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111393+0000", 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2799999999999992e-06 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115386+0000", 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039930820000000002 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115415+0000", 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9408999999999999e-05 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.4s2 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111488+0000", 2024-09-16T04:08:13.713 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.51579407499999996, 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047796619999999996, 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111488+0000", 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111488+0000", 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111488+0000", 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8199999999999999e-07 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111508+0000", 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0550000000000001e-05 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111540+0000", 2024-09-16T04:08:13.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1263000000000002e-05 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111548+0000", 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7690000000000002e-06 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111604+0000", 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5924999999999997e-05 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111613+0000", 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5050000000000007e-06 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116241+0000", 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046280510000000002 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116267+0000", 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6417000000000001e-05 2024-09-16T04:08:13.717 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.718 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.718 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.718 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 37/27 ECSubWrite(tid=129, reqid=osd.3.0:129, at_version=37'43, trim_to=0'0, roll_forward_to=35'41))", 2024-09-16T04:08:13.718 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:13.718 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.51450457699999996, 2024-09-16T04:08:13.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061976449999999999, 2024-09-16T04:08:13.719 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:13.720 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:13.720 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:13.720 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:13.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:13.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:13.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112778+0000", 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3499999999999996e-07 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112797+0000", 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9578000000000001e-05 2024-09-16T04:08:13.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112842+0000", 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5093999999999999e-05 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112850+0000", 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112972+0000", 2024-09-16T04:08:13.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000122033 2024-09-16T04:08:13.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:13.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112982+0000", 2024-09-16T04:08:13.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5789999999999993e-06 2024-09-16T04:08:13.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:13.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118944+0000", 2024-09-16T04:08:13.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005961816 2024-09-16T04:08:13.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:13.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:13.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:13.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118975+0000", 2024-09-16T04:08:13.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0933999999999999e-05 2024-09-16T04:08:13.724 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.724 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.725 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.725 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:13.725 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:13.725 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:13.727 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:13.728 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:13.725+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:13.828 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:13.825+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:13.870 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:13.870 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:13.870 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:13.870 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:13.879 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:13.929 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:13.927+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:13.982 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:13.982 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:13.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:14.032 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:14.030+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:14.105 INFO:tasks.rados.rados.0.smithi016.stdout:115: done (1 left) 2024-09-16T04:08:14.105 INFO:tasks.rados.rados.0.smithi016.stdout:116: done (0 left) 2024-09-16T04:08:14.105 INFO:tasks.rados.rados.0.smithi016.stdout:117: copy_from oid 41 from oid 36 current snap is 5 2024-09-16T04:08:14.106 INFO:tasks.rados.rados.0.smithi016.stdout:118: delete oid 50 current snap is 5 2024-09-16T04:08:14.108 INFO:tasks.rados.rados.0.smithi016.stdout:117: finishing copy_from to smithi01639990-41 2024-09-16T04:08:14.108 INFO:tasks.rados.rados.0.smithi016.stdout:117: got expected ENOENT (src dne) 2024-09-16T04:08:14.108 INFO:tasks.rados.rados.0.smithi016.stdout:117: finishing copy_from racing read to smithi01639990-41 2024-09-16T04:08:14.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:08:14.115 INFO:tasks.rados.rados.0.smithi016.stdout:117: done (1 left) 2024-09-16T04:08:14.116 INFO:tasks.rados.rados.0.smithi016.stdout:118: done (0 left) 2024-09-16T04:08:14.116 INFO:tasks.rados.rados.0.smithi016.stdout:119: snap_create 2024-09-16T04:08:14.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:14.131 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:14.129+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:14.139 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.7s4 32/27 ECSubRead(tid=76, to_read={3:e047da37:::smithi01639990-48:head=0,200704,0}, subchunks={3:e047da37:::smithi01639990-48:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.216154+0000", 2024-09-16T04:08:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.9202838240000002, 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043996529999999999, 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216154+0000", 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216154+0000", 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216155+0000", 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1699999999999997e-07 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216166+0000", 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0966e-05 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216209+0000", 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2670999999999999e-05 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216252+0000", 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3118000000000003e-05 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.216338+0000", 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5654000000000004e-05 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220554+0000", 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042165270000000003 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s7 32/27 ECSubWrite(tid=11, reqid=client.4440.0:199, at_version=32'8, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:08:14.143 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.225161+0000", 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.91127702, 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044456770000000003, 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225161+0000", 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225161+0000", 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225161+0000", 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7800000000000001e-07 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225715+0000", 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00055397300000000003 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225743+0000", 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8034999999999998e-05 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225753+0000", 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1260000000000004e-06 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225864+0000", 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011175699999999999 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.225876+0000", 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1934e-05 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229540+0000", 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036639120000000001 2024-09-16T04:08:14.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229607+0000", 2024-09-16T04:08:14.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6761999999999995e-05 2024-09-16T04:08:14.148 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.148 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.148 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es7 32/27 ECSubWrite(tid=28, reqid=client.4440.0:197, at_version=32'4, trim_to=0'0, roll_forward_to=30'2))", 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.227987+0000", 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.9084511270000002, 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048430519999999996, 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227987+0000", 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227987+0000", 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227989+0000", 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3650000000000001e-06 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228349+0000", 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036041099999999999 2024-09-16T04:08:14.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228398+0000", 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9086999999999999e-05 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228408+0000", 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4420000000000004e-06 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228489+0000", 2024-09-16T04:08:14.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1699000000000005e-05 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228500+0000", 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0516000000000001e-05 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232804+0000", 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043043430000000004 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232830+0000", 2024-09-16T04:08:14.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6188999999999999e-05 2024-09-16T04:08:14.153 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.153 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.153 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.153 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 33/27 ECSubWrite(tid=77, reqid=osd.3.0:77, at_version=33'19, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:14.153 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.095188+0000", 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0412502970000004, 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050055009999999999, 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095188+0000", 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095188+0000", 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095189+0000", 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3499999999999996e-07 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.247e-05 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095237+0000", 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5673999999999998e-05 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095246+0000", 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0170000000000002e-06 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095304+0000", 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8627e-05 2024-09-16T04:08:14.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095316+0000", 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1095e-05 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100160+0000", 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048447230000000004 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100194+0000", 2024-09-16T04:08:14.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3259999999999997e-05 2024-09-16T04:08:14.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.158 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as3 33/27 ECSubWrite(tid=14, reqid=osd.6.0:14, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0412274410000002, 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004947758, 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:14.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0900000000000001e-07 2024-09-16T04:08:14.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095231+0000", 2024-09-16T04:08:14.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9673e-05 2024-09-16T04:08:14.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095274+0000", 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3550000000000001e-05 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095283+0000", 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8740000000000001e-06 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095358+0000", 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4351999999999996e-05 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095366+0000", 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9160000000000007e-06 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100124+0000", 2024-09-16T04:08:14.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047573499999999996 2024-09-16T04:08:14.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100159+0000", 2024-09-16T04:08:14.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4833999999999999e-05 2024-09-16T04:08:14.164 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.164 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s7 33/27 ECSubWrite(tid=13, reqid=osd.4.0:13, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.096050+0000", 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.040388428, 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005993473, 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096050+0000", 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096050+0000", 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096050+0000", 2024-09-16T04:08:14.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7800000000000001e-07 2024-09-16T04:08:14.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096074+0000", 2024-09-16T04:08:14.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3694e-05 2024-09-16T04:08:14.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096103+0000", 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9322000000000001e-05 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096112+0000", 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5939999999999994e-06 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096174+0000", 2024-09-16T04:08:14.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2180000000000004e-05 2024-09-16T04:08:14.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096183+0000", 2024-09-16T04:08:14.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5289999999999997e-06 2024-09-16T04:08:14.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102018+0000", 2024-09-16T04:08:14.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058349539999999998 2024-09-16T04:08:14.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102043+0000", 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5021999999999999e-05 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.102519+0000", 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0339197919999998, 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043264480000000001, 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.173 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102519+0000", 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102519+0000", 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102519+0000", 2024-09-16T04:08:14.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.03e-07 2024-09-16T04:08:14.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102531+0000", 2024-09-16T04:08:14.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2480000000000001e-05 2024-09-16T04:08:14.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102704+0000", 2024-09-16T04:08:14.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017237400000000001 2024-09-16T04:08:14.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102710+0000", 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3090000000000001e-06 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102761+0000", 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1150999999999999e-05 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102769+0000", 2024-09-16T04:08:14.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:08:14.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106808+0000", 2024-09-16T04:08:14.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040384139999999997 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106845+0000", 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7107999999999998e-05 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0336150980000003, 2024-09-16T04:08:14.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083578109999999997, 2024-09-16T04:08:14.181 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.181 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.181 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.181 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102824+0000", 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4899999999999995e-07 2024-09-16T04:08:14.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102859+0000", 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4805999999999998e-05 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102891+0000", 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2394000000000002e-05 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102898+0000", 2024-09-16T04:08:14.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.204e-06 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103849+0000", 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00095089899999999995 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103858+0000", 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3480000000000004e-06 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111149+0000", 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072911809999999999 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111181+0000", 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2329999999999997e-05 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.5s7 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103901+0000", 2024-09-16T04:08:14.185 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0325368770000001, 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045594770000000001, 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103901+0000", 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103901+0000", 2024-09-16T04:08:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103902+0000", 2024-09-16T04:08:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9999999999999999e-07 2024-09-16T04:08:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103927+0000", 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5440999999999999e-05 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103983+0000", 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5951e-05 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103990+0000", 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0729999999999999e-06 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.104050+0000", 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9512000000000002e-05 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.104058+0000", 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.32e-06 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108428+0000", 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043697359999999999 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108461+0000", 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3244000000000002e-05 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.105289+0000", 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0311492649999998, 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065227080000000003, 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.190 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105289+0000", 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105289+0000", 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105290+0000", 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.99e-07 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106024+0000", 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00073392599999999998 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106050+0000", 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6239000000000001e-05 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106058+0000", 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1429999999999998e-06 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106110+0000", 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1788e-05 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106118+0000", 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.225e-06 2024-09-16T04:08:14.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111780+0000", 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056617070000000002 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111812+0000", 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1980999999999998e-05 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.194 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.107403+0000", 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0290357830000003, 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044434460000000002, 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107403+0000", 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107403+0000", 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107403+0000", 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.61e-07 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107537+0000", 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000134163 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107555+0000", 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7833999999999999e-05 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.107560+0000", 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4770000000000002e-06 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108320+0000", 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075933700000000001 2024-09-16T04:08:14.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108329+0000", 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0499999999999997e-06 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111818+0000", 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034895999999999998 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111846+0000", 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7824e-05 2024-09-16T04:08:14.198 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 33/27 ECSubWrite(tid=35, reqid=client.4440.0:212, at_version=33'21, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.118491+0000", 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0179474160000002, 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049879870000000001, 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.199 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118491+0000", 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118491+0000", 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118492+0000", 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118540+0000", 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8677000000000002e-05 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118574+0000", 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3420000000000002e-05 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118583+0000", 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2119999999999992e-06 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118637+0000", 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4243000000000002e-05 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118647+0000", 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0176e-05 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123441+0000", 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047938859999999998 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123479+0000", 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.769e-05 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.204 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141733+0000", 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9947048590000005, 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040667849999999998, 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141733+0000", 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141733+0000", 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141734+0000", 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4000000000000001e-07 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141749+0000", 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.539e-05 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141783+0000", 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3680000000000003e-05 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141791+0000", 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7670000000000003e-06 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141865+0000", 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4352999999999998e-05 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141871+0000", 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9800000000000003e-06 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.145774+0000", 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039023310000000002 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.145800+0000", 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6644000000000001e-05 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.208 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs3 37/27 ECSubWrite(tid=70, reqid=osd.5.0:70, at_version=37'20, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.102767+0000", 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0336710330000001, 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046405389999999999, 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102767+0000", 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102767+0000", 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102767+0000", 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2700000000000001e-07 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102779+0000", 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.204e-05 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102879+0000", 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9389000000000007e-05 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.102885+0000", 2024-09-16T04:08:14.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4659999999999996e-06 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103159+0000", 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00027351399999999998 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103167+0000", 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5019999999999992e-06 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.107378+0000", 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042108329999999998 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.107408+0000", 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9668000000000001e-05 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.213 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 37/27 ECSubWrite(tid=124, reqid=osd.3.0:124, at_version=37'22, trim_to=0'0, roll_forward_to=35'20))", 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.103160+0000", 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0332782149999999, 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042016909999999996, 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103160+0000", 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103160+0000", 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103160+0000", 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.03e-07 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103192+0000", 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1418999999999999e-05 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103242+0000", 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0689999999999997e-05 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103250+0000", 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6660000000000008e-06 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103328+0000", 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8243000000000003e-05 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.103338+0000", 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6739999999999992e-06 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.107323+0000", 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003985438 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.107362+0000", 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8358000000000001e-05 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.217 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.5s7 37/27 ECSubWrite(tid=73, reqid=osd.5.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=37'24))", 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.110813+0000", 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.025625394, 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006035042, 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110813+0000", 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110813+0000", 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110813+0000", 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.44e-07 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110823+0000", 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0529999999999996e-06 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110850+0000", 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7903e-05 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110858+0000", 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0250000000000002e-06 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110903+0000", 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4394999999999998e-05 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110909+0000", 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5150000000000003e-06 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116812+0000", 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005903073 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116848+0000", 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5534000000000002e-05 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.4s4 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111420+0000", 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.02501876, 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053788569999999999, 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.222 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111420+0000", 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111420+0000", 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111420+0000", 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1500000000000001e-07 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111442+0000", 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2195999999999999e-05 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111473+0000", 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0548e-05 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111480+0000", 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6559999999999992e-06 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111529+0000", 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8828000000000001e-05 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111538+0000", 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7199999999999995e-06 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116767+0000", 2024-09-16T04:08:14.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005229666 2024-09-16T04:08:14.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116798+0000", 2024-09-16T04:08:14.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1028000000000002e-05 2024-09-16T04:08:14.227 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.227 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.227 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 37/27 ECSubWrite(tid=128, reqid=osd.3.0:124, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111690+0000", 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.024747925, 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051156070000000003, 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111690+0000", 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111690+0000", 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111691+0000", 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111702+0000", 2024-09-16T04:08:14.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0519e-05 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111730+0000", 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8305000000000001e-05 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111739+0000", 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7530000000000007e-06 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111789+0000", 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9972000000000002e-05 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111797+0000", 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6740000000000003e-06 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116774+0000", 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049763660000000003 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116806+0000", 2024-09-16T04:08:14.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2168000000000001e-05 2024-09-16T04:08:14.232 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.232 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.232 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.232 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs3 37/27 ECSubWrite(tid=74, reqid=osd.5.0:72, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:14.232 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112863+0000", 2024-09-16T04:08:14.232 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0235752810000001, 2024-09-16T04:08:14.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062791490000000004, 2024-09-16T04:08:14.233 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.233 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112863+0000", 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112863+0000", 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112864+0000", 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4900000000000004e-07 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112875+0000", 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1112e-05 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112905+0000", 2024-09-16T04:08:14.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0142999999999999e-05 2024-09-16T04:08:14.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112913+0000", 2024-09-16T04:08:14.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9969999999999998e-06 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112967+0000", 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3928999999999998e-05 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112976+0000", 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.003e-06 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119105+0000", 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061291360000000003 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119142+0000", 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6879999999999999e-05 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.238 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 37/27 ECSubWrite(tid=129, reqid=osd.3.0:129, at_version=37'43, trim_to=0'0, roll_forward_to=35'41))", 2024-09-16T04:08:14.239 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112971+0000", 2024-09-16T04:08:14.239 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.023467135, 2024-09-16T04:08:14.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061749250000000004, 2024-09-16T04:08:14.239 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:14.239 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:14.239 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:14.239 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:14.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:14.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112971+0000", 2024-09-16T04:08:14.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:14.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112971+0000", 2024-09-16T04:08:14.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:14.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:14.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112972+0000", 2024-09-16T04:08:14.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1500000000000004e-07 2024-09-16T04:08:14.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:14.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112992+0000", 2024-09-16T04:08:14.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9856000000000001e-05 2024-09-16T04:08:14.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.113099+0000", 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107217 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.113107+0000", 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1759999999999993e-06 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:14.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.113322+0000", 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021465500000000001 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.113332+0000", 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7059999999999999e-06 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119094+0000", 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057622419999999999 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:14.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:14.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119146+0000", 2024-09-16T04:08:14.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2358000000000003e-05 2024-09-16T04:08:14.245 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.245 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.245 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.245 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:14.245 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:14.245 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:14.248 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:14.249 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:14.245+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:14.352 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:14.350+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:14.382 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:14.382 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:14.395 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:14.395 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:14.403 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:14.404 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:14.405 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:14.423 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:14.424 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:14.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:14.444 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:14.450 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:14.448+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:14.465 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:14.465 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:14.487 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:14.509 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:14.509 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:14.531 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:14.549 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:14.547+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:14.552 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:14.552 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:14.552 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:14.552 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:14.552 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:14.553 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:14.561 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:14.573 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:14.595 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:14.616 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:14.617 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:14.650 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:14.648+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:14.709 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:14.709 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:14.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:14.751 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:14.749+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:14.829 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:14.830 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:14.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:14.884 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:14.882+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:14.948 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:14.948 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:14.952 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:14.951+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:14.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:15.053 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:15.051+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:15.106 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:15.106 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:15.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:08:15.127 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-16T04:08:15.130 INFO:tasks.rados.rados.0.smithi016.stdout:119: done (0 left) 2024-09-16T04:08:15.130 INFO:tasks.rados.rados.0.smithi016.stdout:120: copy_from oid 39 from oid 12 current snap is 6 2024-09-16T04:08:15.130 INFO:tasks.rados.rados.0.smithi016.stdout:121: read oid 2 snap -1 2024-09-16T04:08:15.131 INFO:tasks.rados.rados.0.smithi016.stdout:121: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-16T04:08:15.131 INFO:tasks.rados.rados.0.smithi016.stdout:122: copy_from oid 1 from oid 4 current snap is 6 2024-09-16T04:08:15.131 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 19 current snap is 6 2024-09-16T04:08:15.131 INFO:tasks.rados.rados.0.smithi016.stdout:123: seq_num 76 ranges {0=507904,507904=507904,1015808=262144} 2024-09-16T04:08:15.133 INFO:tasks.rados.rados.0.smithi016.stdout:123: writing smithi01639990-19 from 0 to 507904 tid 1 2024-09-16T04:08:15.137 INFO:tasks.rados.rados.0.smithi016.stdout:123: writing smithi01639990-19 from 507904 to 1015808 tid 2 2024-09-16T04:08:15.139 INFO:tasks.rados.rados.0.smithi016.stdout:123: writing smithi01639990-19 from 1015808 to 1277952 tid 3 2024-09-16T04:08:15.139 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 15 current snap is 6 2024-09-16T04:08:15.139 INFO:tasks.rados.rados.0.smithi016.stdout:124: seq_num 77 ranges {245760=524288,770048=524288,1294336=245760} 2024-09-16T04:08:15.145 INFO:tasks.rados.rados.0.smithi016.stdout:124: writing smithi01639990-15 from 245760 to 770048 tid 1 2024-09-16T04:08:15.149 INFO:tasks.rados.rados.0.smithi016.stdout:124: writing smithi01639990-15 from 770048 to 1294336 tid 2 2024-09-16T04:08:15.150 INFO:tasks.rados.rados.0.smithi016.stdout:124: writing smithi01639990-15 from 1294336 to 1540096 tid 3 2024-09-16T04:08:15.150 INFO:tasks.rados.rados.0.smithi016.stdout:125: read oid 34 snap -1 2024-09-16T04:08:15.151 INFO:tasks.rados.rados.0.smithi016.stdout:125: expect (ObjNum 53 snap 1 seq_num 53) 2024-09-16T04:08:15.151 INFO:tasks.rados.rados.0.smithi016.stdout:126: setattr oid 37 current snap is 6 2024-09-16T04:08:15.151 INFO:tasks.rados.rados.0.smithi016.stdout:121: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-16T04:08:15.152 INFO:tasks.rados.rados.0.smithi016.stdout:122: finishing copy_from racing read to smithi01639990-1 2024-09-16T04:08:15.153 INFO:tasks.rados.rados.0.smithi016.stdout:123: finishing write tid 1 to smithi01639990-19 2024-09-16T04:08:15.153 INFO:tasks.rados.rados.0.smithi016.stdout:122: finishing copy_from to smithi01639990-1 2024-09-16T04:08:15.153 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 1 v 17 (ObjNum 129 snap 0 seq_num 2171441992) dirty exists 2024-09-16T04:08:15.153 INFO:tasks.rados.rados.0.smithi016.stdout:123: finishing write tid 2 to smithi01639990-19 2024-09-16T04:08:15.153 INFO:tasks.rados.rados.0.smithi016.stdout:123: finishing write tid 3 to smithi01639990-19 2024-09-16T04:08:15.153 INFO:tasks.rados.rados.0.smithi016.stdout:124: finishing write tid 1 to smithi01639990-15 2024-09-16T04:08:15.154 INFO:tasks.rados.rados.0.smithi016.stdout:121: done (6 left) 2024-09-16T04:08:15.154 INFO:tasks.rados.rados.0.smithi016.stdout:122: done (5 left) 2024-09-16T04:08:15.154 INFO:tasks.rados.rados.0.smithi016.stdout:127: read oid 22 snap -1 2024-09-16T04:08:15.154 INFO:tasks.rados.rados.0.smithi016.stdout:127: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T04:08:15.154 INFO:tasks.rados.rados.0.smithi016.stdout:128: read oid 31 snap -1 2024-09-16T04:08:15.155 INFO:tasks.rados.rados.0.smithi016.stdout:128: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T04:08:15.155 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 25 current snap is 6 2024-09-16T04:08:15.155 INFO:tasks.rados.rados.0.smithi016.stdout:129: seq_num 78 ranges {2588672=557056,3145728=671744,3817472=98304} 2024-09-16T04:08:15.155 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:15.152+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:15.172 INFO:tasks.rados.rados.0.smithi016.stdout:129: writing smithi01639990-25 from 2588672 to 3145728 tid 1 2024-09-16T04:08:15.179 INFO:tasks.rados.rados.0.smithi016.stdout:129: writing smithi01639990-25 from 3145728 to 3817472 tid 2 2024-09-16T04:08:15.179 INFO:tasks.rados.rados.0.smithi016.stdout:129: writing smithi01639990-25 from 3817472 to 3915776 tid 3 2024-09-16T04:08:15.179 INFO:tasks.rados.rados.0.smithi016.stdout:130: delete oid 44 current snap is 6 2024-09-16T04:08:15.179 INFO:tasks.rados.rados.0.smithi016.stdout:124: finishing write tid 2 to smithi01639990-15 2024-09-16T04:08:15.180 INFO:tasks.rados.rados.0.smithi016.stdout:124: finishing write tid 3 to smithi01639990-15 2024-09-16T04:08:15.180 INFO:tasks.rados.rados.0.smithi016.stdout:124: finishing write tid 4 to smithi01639990-15 2024-09-16T04:08:15.180 INFO:tasks.rados.rados.0.smithi016.stdout:127: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T04:08:15.195 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 37 v 24 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:08:15.196 INFO:tasks.rados.rados.0.smithi016.stdout:128: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T04:08:15.227 INFO:tasks.rados.rados.0.smithi016.stdout:124: finishing write tid 5 to smithi01639990-15 2024-09-16T04:08:15.228 INFO:tasks.rados.rados.0.smithi016.stdout:124: oid 15 updating version 0 to 47 2024-09-16T04:08:15.228 INFO:tasks.rados.rados.0.smithi016.stdout:124: oid 15 version 47 is already newer than 46 2024-09-16T04:08:15.228 INFO:tasks.rados.rados.0.smithi016.stdout:124: oid 15 version 47 is already newer than 45 2024-09-16T04:08:15.228 INFO:tasks.rados.rados.0.smithi016.stdout:124: oid 15 updating version 47 to 48 2024-09-16T04:08:15.228 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 15 v 48 (ObjNum 77 snap 6 seq_num 77) dirty exists 2024-09-16T04:08:15.228 INFO:tasks.rados.rados.0.smithi016.stdout:124: left oid 15 (ObjNum 77 snap 6 seq_num 77) 2024-09-16T04:08:15.228 INFO:tasks.rados.rados.0.smithi016.stdout:124: done (8 left) 2024-09-16T04:08:15.228 INFO:tasks.rados.rados.0.smithi016.stdout:126: done (7 left) 2024-09-16T04:08:15.228 INFO:tasks.rados.rados.0.smithi016.stdout:127: done (6 left) 2024-09-16T04:08:15.229 INFO:tasks.rados.rados.0.smithi016.stdout:128: done (5 left) 2024-09-16T04:08:15.229 INFO:tasks.rados.rados.0.smithi016.stdout:130: done (4 left) 2024-09-16T04:08:15.229 INFO:tasks.rados.rados.0.smithi016.stdout:131: snap_create 2024-09-16T04:08:15.229 INFO:tasks.rados.rados.0.smithi016.stdout:129: finishing write tid 1 to smithi01639990-25 2024-09-16T04:08:15.229 INFO:tasks.rados.rados.0.smithi016.stdout:129: finishing write tid 2 to smithi01639990-25 2024-09-16T04:08:15.253 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:15.253 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:15.254 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:15.253+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:15.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:08:15.339 INFO:tasks.rados.rados.0.smithi016.stdout:129: finishing write tid 3 to smithi01639990-25 2024-09-16T04:08:15.341 INFO:tasks.rados.rados.0.smithi016.stdout:120: finishing copy_from racing read to smithi01639990-39 2024-09-16T04:08:15.342 INFO:tasks.rados.rados.0.smithi016.stdout:120: finishing copy_from to smithi01639990-39 2024-09-16T04:08:15.342 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 39 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T04:08:15.355 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:15.353+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:15.410 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:15.410 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:15.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:08:15.429 INFO:tasks.rados.rados.0.smithi016.stdout:120: done (4 left) 2024-09-16T04:08:15.429 INFO:tasks.rados.rados.0.smithi016.stdout:131: done (3 left) 2024-09-16T04:08:15.430 INFO:tasks.rados.rados.0.smithi016.stdout:132: read oid 46 snap -1 2024-09-16T04:08:15.430 INFO:tasks.rados.rados.0.smithi016.stdout:132: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-16T04:08:15.430 INFO:tasks.rados.rados.0.smithi016.stdout:133: read oid 27 snap -1 2024-09-16T04:08:15.430 INFO:tasks.rados.rados.0.smithi016.stdout:133: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T04:08:15.430 INFO:tasks.rados.rados.0.smithi016.stdout:134: snap_remove snap 3 2024-09-16T04:08:15.474 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:15.472+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:15.540 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:15.541 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:15.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:15.557 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:15.555+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:15.658 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:15.656+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:15.697 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:15.698 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:15.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:15.758 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:15.757+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:15.855 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:15.855 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:15.859 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:15.858+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:15.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:15.961 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:15.959+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:15.972 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:15.972 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:15.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:16.062 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:16.059+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:16.086 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:16.086 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:16.095 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:16.163 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:16.162+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:16.202 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:16.202 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:16.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:16.264 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:16.262+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:16.318 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:16.318 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:16.326 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:16.365 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:16.364+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:16.431 INFO:tasks.rados.rados.0.smithi016.stdout:134: done (5 left) 2024-09-16T04:08:16.431 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 26 current snap is 7 2024-09-16T04:08:16.431 INFO:tasks.rados.rados.0.smithi016.stdout:135: seq_num 80 ranges {425984=589824,1015808=573440,1589248=196608} 2024-09-16T04:08:16.438 INFO:tasks.rados.rados.0.smithi016.stdout:135: writing smithi01639990-26 from 425984 to 1015808 tid 1 2024-09-16T04:08:16.442 INFO:tasks.rados.rados.0.smithi016.stdout:135: writing smithi01639990-26 from 1015808 to 1589248 tid 2 2024-09-16T04:08:16.443 INFO:tasks.rados.rados.0.smithi016.stdout:135: writing smithi01639990-26 from 1589248 to 1785856 tid 3 2024-09-16T04:08:16.443 INFO:tasks.rados.rados.0.smithi016.stdout:129: finishing write tid 4 to smithi01639990-25 2024-09-16T04:08:16.443 INFO:tasks.rados.rados.0.smithi016.stdout:136: snap_remove snap 5 2024-09-16T04:08:16.451 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:16.451 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:16.460 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:16.504 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:16.502+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:16.563 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:16.563 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:16.566 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:16.565+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:16.572 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:16.628 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:16.667 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:16.665+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:16.715 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:16.716 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:16.724 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:16.762 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:16.762 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:16.767 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:16.766+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:16.783 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:16.783 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:16.803 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:16.803 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:16.868 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:16.869 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:16.869 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:16.866+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:16.877 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:16.969 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:16.967+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:17.021 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:17.021 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:17.025 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:17.025 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:17.029 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:17.046 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:17.047 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:17.068 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:17.068 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:17.075 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:17.072+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:17.090 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:17.111 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:17.111 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:17.133 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:17.136 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:17.136 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:17.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:17.136 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:17.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.136 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4209.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-09-16T04:08:17.136 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.939368+0000", 2024-09-16T04:08:17.136 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.194058664, 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060687750000000002, 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4209", 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/1971921389", 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 7 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.939368+0000", 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.939368+0000", 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.939380+0000", 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2094e-05 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.939398+0000", 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8165999999999999e-05 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.939429+0000", 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0555999999999997e-05 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.939447+0000", 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.853e-05 2024-09-16T04:08:17.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.939635+0000", 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000187505 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.939646+0000", 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0862999999999999e-05 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.943447+0000", 2024-09-16T04:08:17.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038013439999999999 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.943516+0000", 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8931999999999997e-05 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 1", 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.943613+0000", 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6506000000000001e-05 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:17.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.945367+0000", 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001754132 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.945379+0000", 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2174999999999999e-05 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.945413+0000", 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4105e-05 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.945437+0000", 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3867e-05 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:184 3.5s0 3:a7f99973:::smithi01639990-25:head [setxattr _header (56) in=7b] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.185063+0000", 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.9483632740000001, 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065214890000000001, 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.143 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 184 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185063+0000", 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185063+0000", 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185064+0000", 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7300000000000005e-07 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185068+0000", 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8550000000000004e-06 2024-09-16T04:08:17.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185079+0000", 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0674e-05 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185085+0000", 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3409999999999999e-06 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.186974+0000", 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0018891559999999999 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187064+0000", 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0127e-05 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.187428+0000", 2024-09-16T04:08:17.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000363779 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.188756+0000", 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013281390000000001 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191512+0000", 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027552729999999999 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191585+0000", 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3372000000000002e-05 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:185 3.5s0 3:a7f99973:::smithi01639990-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.185094+0000", 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.9483322950000002, 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077979190000000004, 2024-09-16T04:08:17.149 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 185 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185094+0000", 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185094+0000", 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185095+0000", 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7299999999999996e-07 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185098+0000", 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4679999999999999e-06 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185104+0000", 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5189999999999998e-06 2024-09-16T04:08:17.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185109+0000", 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.057e-06 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.188386+0000", 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032768250000000001 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:17.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.188443+0000", 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7426000000000002e-05 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191927+0000", 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034834190000000002 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.192006+0000", 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9634000000000004e-05 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.192892+0000", 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088599800000000004 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s5 33/27 ECSubWrite(tid=13, reqid=osd.4.0:13, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.095950+0000", 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0374771379999999, 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0064684540000000002, 2024-09-16T04:08:17.155 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095950+0000", 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095950+0000", 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095950+0000", 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5900000000000001e-07 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.095978+0000", 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8031e-05 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.096505+0000", 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00052695900000000002 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.096515+0000", 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0165e-05 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.096577+0000", 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1915000000000006e-05 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.096586+0000", 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2550000000000003e-06 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102382+0000", 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057955460000000004 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102418+0000", 2024-09-16T04:08:17.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6424000000000002e-05 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 33/27 ECSubWrite(tid=37, reqid=osd.8.0:37, at_version=33'25, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0347200609999998, 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088041119999999994, 2024-09-16T04:08:17.160 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7000000000000001e-07 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098728+0000", 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1432000000000001e-05 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098770+0000", 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1486999999999998e-05 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098780+0000", 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7280000000000007e-06 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098852+0000", 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2461999999999998e-05 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098862+0000", 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9369999999999999e-06 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107472+0000", 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086102990000000001 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107511+0000", 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8497000000000001e-05 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0335494490000006, 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011088002, 2024-09-16T04:08:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099878+0000", 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8400000000000003e-07 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099902+0000", 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3629999999999999e-05 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099934+0000", 2024-09-16T04:08:17.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2388999999999998e-05 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099943+0000", 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4800000000000001e-06 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.100212+0000", 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026930999999999998 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.100220+0000", 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8339999999999999e-06 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110928+0000", 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010708333 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110965+0000", 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7141999999999998e-05 2024-09-16T04:08:17.169 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s7 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0313371290000006, 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088092050000000005, 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.18e-07 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102106+0000", 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6116999999999998e-05 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102134+0000", 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7963e-05 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102140+0000", 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102224+0000", 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3634000000000006e-05 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102233+0000", 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1609999999999992e-06 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110868+0000", 2024-09-16T04:08:17.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086352040000000005 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110899+0000", 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0852999999999998e-05 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0304112940000003, 2024-09-16T04:08:17.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013925085, 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103016+0000", 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5499999999999999e-07 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103038+0000", 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2416e-05 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103157+0000", 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000118439 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103166+0000", 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9819999999999997e-06 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103225+0000", 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8996000000000002e-05 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103235+0000", 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9909999999999997e-06 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116899+0000", 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013664549999999999 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116941+0000", 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1455999999999997e-05 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0303859929999994, 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013929731000000001, 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5899999999999998e-07 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103057+0000", 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5778e-05 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103193+0000", 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0001359 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103199+0000", 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1279999999999996e-06 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107717+0000", 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045183769999999996 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107726+0000", 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5420000000000005e-06 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116950+0000", 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092240229999999996 2024-09-16T04:08:17.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116970+0000", 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0724e-05 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s5 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0296371910000008, 2024-09-16T04:08:17.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070187460000000002, 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103801+0000", 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0927999999999999e-05 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103842+0000", 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0649999999999999e-05 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103851+0000", 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8920000000000006e-06 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103944+0000", 2024-09-16T04:08:17.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2705000000000001e-05 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103955+0000", 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1282e-05 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110771+0000", 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0068159270000000003 2024-09-16T04:08:17.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110808+0000", 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7512000000000003e-05 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(osd.3.19:2 3.cs0 3:3d7a8735:::smithi01639990-20:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105118+0000", 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0283090389999998, 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0066834290000000003, 2024-09-16T04:08:17.189 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105118+0000", 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105118+0000", 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105120+0000", 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.855e-06 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105129+0000", 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2739999999999996e-06 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105164+0000", 2024-09-16T04:08:17.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5621000000000003e-05 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105182+0000", 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7790000000000001e-05 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107894+0000", 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027115860000000002 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107957+0000", 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3202999999999998e-05 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111801+0000", 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038441 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.193 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/27 ECSubWriteReply(tid=39, last_complete=33'12, committed=1, applied=1))", 2024-09-16T04:08:17.194 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105528+0000", 2024-09-16T04:08:17.194 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0278991029999993, 2024-09-16T04:08:17.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060053179999999999, 2024-09-16T04:08:17.194 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.194 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.194 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.194 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105528+0000", 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105528+0000", 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105528+0000", 2024-09-16T04:08:17.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7999999999999995e-07 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105535+0000", 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.092e-06 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105560+0000", 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4635e-05 2024-09-16T04:08:17.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105565+0000", 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1259999999999997e-06 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111497+0000", 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059320049999999997 2024-09-16T04:08:17.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111533+0000", 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.578e-05 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.200 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 33/27 ECSubWriteReply(tid=39, last_complete=33'12, committed=1, applied=1))", 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105863+0000", 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.0275638429999994, 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057196360000000002, 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105863+0000", 2024-09-16T04:08:17.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105863+0000", 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105863+0000", 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7999999999999995e-07 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105869+0000", 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6300000000000003e-06 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105887+0000", 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8300000000000001e-05 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105892+0000", 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6480000000000002e-06 2024-09-16T04:08:17.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111563+0000", 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0056711579999999999 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111583+0000", 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9320000000000001e-05 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=34, reqid=client.4440.0:211, at_version=33'20, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.115669+0000", 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.017758121, 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0059919760000000004, 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.205 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115669+0000", 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115669+0000", 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115669+0000", 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8700000000000005e-07 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115996+0000", 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00032679699999999998 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116023+0000", 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6389000000000001e-05 2024-09-16T04:08:17.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116028+0000", 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8329999999999999e-06 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116074+0000", 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5383999999999998e-05 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116080+0000", 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9050000000000002e-06 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121623+0000", 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0055436799999999996 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121661+0000", 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7301000000000001e-05 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 33/27 ECSubWrite(tid=85, reqid=client.4440.0:216, at_version=33'15, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:17.210 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.140827+0000", 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 7.9925996020000003, 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069260850000000002, 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140827+0000", 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140827+0000", 2024-09-16T04:08:17.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140828+0000", 2024-09-16T04:08:17.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1900000000000001e-07 2024-09-16T04:08:17.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141044+0000", 2024-09-16T04:08:17.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021635099999999999 2024-09-16T04:08:17.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141081+0000", 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6628000000000002e-05 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141087+0000", 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.297e-06 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141146+0000", 2024-09-16T04:08:17.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8838999999999997e-05 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141157+0000", 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1105e-05 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.147715+0000", 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065581980000000003 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.147753+0000", 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7747999999999998e-05 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.141651+0000", 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 7.9917759190000002, 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0064293409999999999, 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.216 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141651+0000", 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141651+0000", 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141652+0000", 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8700000000000005e-07 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141666+0000", 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4146e-05 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141691+0000", 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5329000000000001e-05 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141697+0000", 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0410000000000003e-06 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142751+0000", 2024-09-16T04:08:17.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010543880000000001 2024-09-16T04:08:17.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142761+0000", 2024-09-16T04:08:17.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1449999999999997e-06 2024-09-16T04:08:17.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.148051+0000", 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.005290182 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.148080+0000", 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9422999999999999e-05 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s5 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0265223030000001, 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0080941920000000001, 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106905+0000", 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4999999999999999e-07 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106926+0000", 2024-09-16T04:08:17.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1202000000000001e-05 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107015+0000", 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9407999999999995e-05 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107034+0000", 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8896999999999999e-05 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107499+0000", 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046474800000000002 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107509+0000", 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0390999999999999e-05 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114906+0000", 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073969409999999998 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114999+0000", 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2454999999999995e-05 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds7 37/31 ECSubWrite(tid=163, reqid=osd.7.0:163, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:17.226 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0248691000000001, 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084527260000000007, 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0100000000000004e-07 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108578+0000", 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9366999999999999e-05 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108616+0000", 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8056999999999998e-05 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108624+0000", 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.824e-06 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108684+0000", 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0019999999999998e-05 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108693+0000", 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2599999999999994e-06 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116974+0000", 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082815990000000006 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.117010+0000", 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5898000000000001e-05 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds7 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0238951959999998, 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091377019999999993, 2024-09-16T04:08:17.231 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6499999999999998e-07 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109554+0000", 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2030000000000001e-05 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109587+0000", 2024-09-16T04:08:17.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3019999999999999e-05 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109603+0000", 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5678000000000001e-05 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110514+0000", 2024-09-16T04:08:17.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00091074700000000005 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110523+0000", 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4639999999999995e-06 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118636+0000", 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008112157 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118669+0000", 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3741000000000001e-05 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0220121689999999, 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0072197169999999996, 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.236 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7400000000000003e-07 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111431+0000", 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5305e-05 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111457+0000", 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6067e-05 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111462+0000", 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3480000000000003e-06 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111516+0000", 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3897000000000003e-05 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111526+0000", 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7440000000000002e-06 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118592+0000", 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070660710000000002 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118634+0000", 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2611000000000002e-05 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.240 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:17.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:08:17.242 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:17.243 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:17.243 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:17.244 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:17.240+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:17.301 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:17.301 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:17.338 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:17.338 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:17.343 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:17.342+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:17.388 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:17.389 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:17.389 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:17.389 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:17.398 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:17.434 INFO:tasks.rados.rados.0.smithi016.stdout:136: done (6 left) 2024-09-16T04:08:17.434 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 29 current snap is 7 2024-09-16T04:08:17.434 INFO:tasks.rados.rados.0.smithi016.stdout:123: finishing write tid 4 to smithi01639990-19 2024-09-16T04:08:17.434 INFO:tasks.rados.rados.0.smithi016.stdout:137: seq_num 81 ranges {475136=606208,1081344=720896,1802240=49152} 2024-09-16T04:08:17.441 INFO:tasks.rados.rados.0.smithi016.stdout:137: writing smithi01639990-29 from 475136 to 1081344 tid 1 2024-09-16T04:08:17.444 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:17.441+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:17.446 INFO:tasks.rados.rados.0.smithi016.stdout:137: writing smithi01639990-29 from 1081344 to 1802240 tid 2 2024-09-16T04:08:17.447 INFO:tasks.rados.rados.0.smithi016.stdout:137: writing smithi01639990-29 from 1802240 to 1851392 tid 3 2024-09-16T04:08:17.447 INFO:tasks.rados.rados.0.smithi016.stdout:138: read oid 33 snap 4 2024-09-16T04:08:17.448 INFO:tasks.rados.rados.0.smithi016.stdout:138: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:08:17.448 INFO:tasks.rados.rados.0.smithi016.stdout:123: finishing write tid 5 to smithi01639990-19 2024-09-16T04:08:17.448 INFO:tasks.rados.rados.0.smithi016.stdout:123: oid 19 updating version 0 to 17 2024-09-16T04:08:17.448 INFO:tasks.rados.rados.0.smithi016.stdout:123: oid 19 version 17 is already newer than 15 2024-09-16T04:08:17.448 INFO:tasks.rados.rados.0.smithi016.stdout:123: oid 19 version 17 is already newer than 14 2024-09-16T04:08:17.448 INFO:tasks.rados.rados.0.smithi016.stdout:123: oid 19 version 17 is already newer than 16 2024-09-16T04:08:17.448 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 19 v 17 (ObjNum 76 snap 6 seq_num 76) dirty exists 2024-09-16T04:08:17.448 INFO:tasks.rados.rados.0.smithi016.stdout:123: left oid 19 (ObjNum 76 snap 6 seq_num 76) 2024-09-16T04:08:17.448 INFO:tasks.rados.rados.0.smithi016.stdout:133: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T04:08:17.464 INFO:tasks.rados.rados.0.smithi016.stdout:132: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-16T04:08:17.484 INFO:tasks.rados.rados.0.smithi016.stdout:125: expect (ObjNum 53 snap 1 seq_num 53) 2024-09-16T04:08:17.510 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:17.510 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:17.510 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:17.519 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:17.545 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:17.543+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:17.576 INFO:tasks.rados.rados.0.smithi016.stdout:129: finishing write tid 5 to smithi01639990-25 2024-09-16T04:08:17.576 INFO:tasks.rados.rados.0.smithi016.stdout:129: oid 25 updating version 0 to 30 2024-09-16T04:08:17.576 INFO:tasks.rados.rados.0.smithi016.stdout:129: oid 25 version 30 is already newer than 28 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:129: oid 25 version 30 is already newer than 29 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:129: oid 25 updating version 30 to 31 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 25 v 31 (ObjNum 78 snap 6 seq_num 78) dirty exists 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:129: left oid 25 (ObjNum 78 snap 6 seq_num 78) 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:123: done (7 left) 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:125: done (6 left) 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:129: done (5 left) 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:132: done (4 left) 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:133: done (3 left) 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 46 current snap is 7 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:135: finishing write tid 1 to smithi01639990-26 2024-09-16T04:08:17.577 INFO:tasks.rados.rados.0.smithi016.stdout:135: finishing write tid 2 to smithi01639990-26 2024-09-16T04:08:17.578 INFO:tasks.rados.rados.0.smithi016.stdout:139: seq_num 82 ranges {770048=622592,1392640=475136,1867776=294912} 2024-09-16T04:08:17.583 INFO:tasks.rados.rados.0.smithi016.stdout:139: writing smithi01639990-46 from 770048 to 1392640 tid 1 2024-09-16T04:08:17.586 INFO:tasks.rados.rados.0.smithi016.stdout:139: writing smithi01639990-46 from 1392640 to 1867776 tid 2 2024-09-16T04:08:17.588 INFO:tasks.rados.rados.0.smithi016.stdout:139: writing smithi01639990-46 from 1867776 to 2162688 tid 3 2024-09-16T04:08:17.588 INFO:tasks.rados.rados.0.smithi016.stdout:140: read oid 47 snap -1 2024-09-16T04:08:17.589 INFO:tasks.rados.rados.0.smithi016.stdout:140: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T04:08:17.589 INFO:tasks.rados.rados.0.smithi016.stdout:141: snap_remove snap 6 2024-09-16T04:08:17.646 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:17.643+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:193 3.as0 3:5d02464a:::smithi01639990-40:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4665933399999993, 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041575429999999997, 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.666 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 193 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196589+0000", 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8160000000000001e-06 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196600+0000", 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1545e-05 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196613+0000", 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.34e-05 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196621+0000", 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4909999999999999e-06 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196682+0000", 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1074000000000005e-05 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196776+0000", 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4102000000000001e-05 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.197563+0000", 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000786446 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.198374+0000", 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081128199999999998 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200668+0000", 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0022936979999999998 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200744+0000", 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6688999999999997e-05 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.671 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.5594766609999997, 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0068480019999999997, 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.672 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103704+0000", 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7100000000000001e-07 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103727+0000", 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3694999999999999e-05 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103810+0000", 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3091999999999997e-05 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103816+0000", 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4519999999999996e-06 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103971+0000", 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000154993 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103982+0000", 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0933e-05 2024-09-16T04:08:17.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110491+0000", 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065096800000000003 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110551+0000", 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9985999999999998e-05 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.5594273229999995, 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067325809999999996, 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103754+0000", 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5899999999999995e-07 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103768+0000", 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4562999999999999e-05 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103805+0000", 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6730000000000002e-05 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103813+0000", 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103868+0000", 2024-09-16T04:08:17.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4475000000000003e-05 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103877+0000", 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7099999999999996e-06 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110444+0000", 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065675170000000001 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110485+0000", 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1177000000000002e-05 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/27 ECSubWrite(tid=84, reqid=client.4440.0:215, at_version=33'14, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.5244677039999992, 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042300860000000001, 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.681 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5900000000000001e-07 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139362+0000", 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064961800000000003 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139387+0000", 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5369999999999999e-05 2024-09-16T04:08:17.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139394+0000", 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2190000000000001e-06 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139458+0000", 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4603000000000005e-05 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139470+0000", 2024-09-16T04:08:17.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1810999999999999e-05 2024-09-16T04:08:17.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142910+0000", 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003439896 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142942+0000", 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2410000000000003e-05 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/27 ECSubWrite(tid=85, reqid=client.4440.0:216, at_version=33'15, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.5222654309999992, 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043866510000000001, 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:17.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141225+0000", 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00030972200000000001 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141258+0000", 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3228000000000001e-05 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141267+0000", 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5590000000000006e-06 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141327+0000", 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0100999999999999e-05 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141338+0000", 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0745e-05 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145260+0000", 2024-09-16T04:08:17.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039227610000000003 2024-09-16T04:08:17.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145301+0000", 2024-09-16T04:08:17.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0884999999999999e-05 2024-09-16T04:08:17.691 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 37/27 ECSubWrite(tid=71, reqid=osd.5.0:71, at_version=37'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5604895120000002, 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040595509999999998, 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:17.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4000000000000001e-07 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102711+0000", 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0409999999999999e-05 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103362+0000", 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065125100000000004 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103372+0000", 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7340000000000003e-06 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103464+0000", 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1663999999999995e-05 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103474+0000", 2024-09-16T04:08:17.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8630000000000003e-06 2024-09-16T04:08:17.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106711+0000", 2024-09-16T04:08:17.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032372669999999998 2024-09-16T04:08:17.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106750+0000", 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9221999999999998e-05 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=70, reqid=osd.5.0:70, at_version=37'20, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5604673909999995, 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0053540310000000004, 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.697 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3799999999999999e-07 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102721+0000", 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6500000000000002e-06 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103774+0000", 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010529070000000001 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103806+0000", 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1303999999999997e-05 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103868+0000", 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1937e-05 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103877+0000", 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6199999999999994e-06 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108037+0000", 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041593389999999997 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108067+0000", 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0136000000000001e-05 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/27 ECSubWrite(tid=124, reqid=osd.3.0:124, at_version=37'22, trim_to=0'0, roll_forward_to=35'20))", 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.559843023, 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070925160000000001, 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6999999999999999e-07 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103359+0000", 2024-09-16T04:08:17.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1835999999999999e-05 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103683+0000", 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00032427399999999999 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103693+0000", 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7380000000000006e-06 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103756+0000", 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3115999999999997e-05 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103766+0000", 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0043e-05 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110407+0000", 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0066407230000000003 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110430+0000", 2024-09-16T04:08:17.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2616000000000002e-05 2024-09-16T04:08:17.706 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.706 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.706 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5590127499999999, 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008202661, 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104168+0000", 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0200000000000001e-07 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104188+0000", 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9933999999999999e-05 2024-09-16T04:08:17.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104232+0000", 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4196000000000002e-05 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104241+0000", 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7979999999999995e-06 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104928+0000", 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687124 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104937+0000", 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5269999999999998e-06 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112328+0000", 2024-09-16T04:08:17.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073914510000000003 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112370+0000", 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1928999999999998e-05 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.bs7 37/27 ECSubWrite(tid=127, reqid=osd.3.0:127, at_version=37'22, trim_to=0'0, roll_forward_to=35'17))", 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.105041+0000", 2024-09-16T04:08:17.711 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5581388619999998, 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046635560000000001, 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105041+0000", 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105041+0000", 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105042+0000", 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9299999999999997e-07 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105073+0000", 2024-09-16T04:08:17.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1041999999999999e-05 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105110+0000", 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7113000000000002e-05 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105118+0000", 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4330000000000004e-06 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105253+0000", 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000134191 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105264+0000", 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1827e-05 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109656+0000", 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043918459999999996 2024-09-16T04:08:17.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109705+0000", 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8411000000000001e-05 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5562931420000004, 2024-09-16T04:08:17.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082659529999999995, 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:17.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1600000000000002e-07 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106910+0000", 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2954000000000001e-05 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106990+0000", 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9788000000000002e-05 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106997+0000", 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4379999999999998e-06 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107267+0000", 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026999599999999999 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107276+0000", 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9530000000000005e-06 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115126+0000", 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078491940000000003 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115153+0000", 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7214000000000001e-05 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.720 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 37/27 ECSubWrite(tid=73, reqid=osd.5.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=37'24))", 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.110673+0000", 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5525069900000004, 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044773169999999998, 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110673+0000", 2024-09-16T04:08:17.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110673+0000", 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110674+0000", 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6199999999999997e-07 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110685+0000", 2024-09-16T04:08:17.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1221e-05 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110720+0000", 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5049000000000003e-05 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110729+0000", 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3370000000000006e-06 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110800+0000", 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0425000000000004e-05 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110810+0000", 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-05 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115120+0000", 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043095920000000001 2024-09-16T04:08:17.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115150+0000", 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0840999999999998e-05 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.4s6 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5518071730000003, 2024-09-16T04:08:17.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0054036320000000002, 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899999999999999e-07 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111387+0000", 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3501e-05 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111416+0000", 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9377999999999999e-05 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111421+0000", 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2889999999999997e-06 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111470+0000", 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9162000000000001e-05 2024-09-16T04:08:17.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111479+0000", 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9900000000000003e-06 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116730+0000", 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052506219999999999 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116777+0000", 2024-09-16T04:08:17.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6551000000000002e-05 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/27 ECSubWrite(tid=128, reqid=osd.3.0:124, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.111572+0000", 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5516079759999997, 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038855840000000001, 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111572+0000", 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111572+0000", 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111573+0000", 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:17.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111582+0000", 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3649999999999993e-06 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111616+0000", 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4480000000000002e-05 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111624+0000", 2024-09-16T04:08:17.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7160000000000003e-06 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111676+0000", 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1742e-05 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111682+0000", 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7790000000000001e-06 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115422+0000", 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003740307 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115458+0000", 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5639999999999998e-05 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=74, reqid=osd.5.0:72, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5503803640000005, 2024-09-16T04:08:17.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040065320000000001, 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5400000000000001e-07 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112809+0000", 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9050000000000003e-06 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112841+0000", 2024-09-16T04:08:17.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1983999999999997e-05 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112849+0000", 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3750000000000003e-06 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112915+0000", 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5599000000000007e-05 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112922+0000", 2024-09-16T04:08:17.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116782+0000", 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038605890000000002 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116806+0000", 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3760999999999998e-05 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.213239917, 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091491550000000008, 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5800000000000001e-07 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450512+0000", 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057160000000000002 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450541+0000", 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8835000000000001e-05 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450549+0000", 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3080000000000008e-06 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450607+0000", 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7763999999999998e-05 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450616+0000", 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459045+0000", 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084290709999999998 2024-09-16T04:08:17.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459089+0000", 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4233000000000002e-05 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/27 ECSubWrite(tid=41, reqid=osd.0.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.451624+0000", 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.211555719, 2024-09-16T04:08:17.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044678180000000001, 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451624+0000", 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451624+0000", 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451625+0000", 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7100000000000001e-07 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451635+0000", 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.825e-06 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451672+0000", 2024-09-16T04:08:17.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7134000000000001e-05 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451680+0000", 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451737+0000", 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6675000000000002e-05 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451744+0000", 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.041e-06 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456050+0000", 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043059860000000004 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456092+0000", 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2014999999999997e-05 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.747 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.748 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=160, reqid=client.4440.0:347, at_version=42'32, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:17.748 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:17.748 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.210617835, 2024-09-16T04:08:17.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065520760000000004, 2024-09-16T04:08:17.748 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.748 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.748 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452563+0000", 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6300000000000005e-07 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452621+0000", 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8488e-05 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452654+0000", 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2750999999999999e-05 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452659+0000", 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.293e-06 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452730+0000", 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0173999999999996e-05 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452738+0000", 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6260000000000001e-06 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459095+0000", 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0063566530000000003 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459114+0000", 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9428000000000001e-05 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.752 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=161, reqid=client.4440.0:348, at_version=42'33, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.21055522900000001, 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065540390000000002, 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6300000000000001e-07 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452632+0000", 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6699999999999997e-06 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452678+0000", 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5760000000000002e-05 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452682+0000", 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.036e-06 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453280+0000", 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059861699999999996 2024-09-16T04:08:17.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453289+0000", 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9590000000000001e-06 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459117+0000", 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058281770000000004 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459179+0000", 2024-09-16T04:08:17.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1556999999999994e-05 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=167, reqid=client.4440.0:355, at_version=42'37, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.588912+0000", 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.074267674000000006, 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004899128, 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:17.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.588912+0000", 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.588912+0000", 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.588913+0000", 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4199999999999995e-07 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589099+0000", 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018556200000000001 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589134+0000", 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5553999999999999e-05 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589142+0000", 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0979999999999994e-06 2024-09-16T04:08:17.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589199+0000", 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6586999999999999e-05 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589210+0000", 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.078e-05 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.593773+0000", 2024-09-16T04:08:17.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045635179999999999 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.593811+0000", 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8287000000000001e-05 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:17.761 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:17.763 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:17.763 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:17.760+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:17.863 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:17.862+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:17.907 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:17.907 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:17.907 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:17.908 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:17.917 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:17.964 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:17.963+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:18.021 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:18.021 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:18.021 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:18.021 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:18.021 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:18.027 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:18.065 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:18.063+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:18.132 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:18.132 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:18.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:18.132 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:18.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.132 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:18.132 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:18.132 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.188537933999999, 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:18.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:18.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:18.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:18.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:18.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:18.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:18.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 27.206133691000002, 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:18.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:18.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 27.195771185000002, 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:18.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:18.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.699183132999998, 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.150 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:18.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:18.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.675670231000002, 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092583400000000003, 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.37e-07 2024-09-16T04:08:18.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453604+0000", 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2658e-05 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453611+0000", 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0650000000000001e-06 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453616+0000", 2024-09-16T04:08:18.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9099999999999996e-06 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459323+0000", 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057072879999999996 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459369+0000", 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6436000000000001e-05 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459442+0000", 2024-09-16T04:08:18.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2459000000000005e-05 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460087+0000", 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064517999999999997 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462796+0000", 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027087639999999998 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462827+0000", 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0827000000000001e-05 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462849+0000", 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2215999999999999e-05 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.161 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.244874041999999, 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088271749999999996, 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 158 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884387+0000", 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5899999999999996e-07 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884453+0000", 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999997e-05 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884462+0000", 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2920000000000001e-06 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884468+0000", 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9399999999999999e-06 2024-09-16T04:08:18.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886225+0000", 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0017572079999999999 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886269+0000", 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3118999999999998e-05 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.886334+0000", 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5695e-05 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888544+0000", 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002210086 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893152+0000", 2024-09-16T04:08:18.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046078940000000004 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893177+0000", 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.463e-05 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893214+0000", 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7002000000000002e-05 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.244779012999999, 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087992620000000004, 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 159 2024-09-16T04:08:18.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884482+0000", 2024-09-16T04:08:18.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0500000000000003e-07 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884486+0000", 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8299999999999998e-06 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884493+0000", 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8759999999999999e-06 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884498+0000", 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3550000000000004e-06 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888576+0000", 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040780230000000001 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888629+0000", 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2892000000000001e-05 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888704+0000", 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5024000000000005e-05 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892724+0000", 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040194619999999997 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893222+0000", 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049801700000000001 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893246+0000", 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3901e-05 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893281+0000", 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5277000000000001e-05 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:18.173 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.244750977999999, 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088385849999999995, 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 160 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3900000000000005e-07 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884514+0000", 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2169999999999999e-06 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884520+0000", 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8639999999999998e-06 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884527+0000", 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.866e-06 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888915+0000", 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043874270000000002 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888953+0000", 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7839000000000001e-05 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889014+0000", 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1289000000000002e-05 2024-09-16T04:08:18.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892771+0000", 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037575199999999999 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893289+0000", 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00051718599999999999 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893314+0000", 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5202999999999999e-05 2024-09-16T04:08:18.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893348+0000", 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4635000000000003e-05 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.244713797999999, 2024-09-16T04:08:18.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088608530000000001, 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 161 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884548+0000", 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5599999999999995e-07 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884603+0000", 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5615999999999997e-05 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884612+0000", 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6589999999999996e-06 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884619+0000", 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4630000000000004e-06 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889203+0000", 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045841989999999997 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889242+0000", 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8713e-05 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889304+0000", 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1858000000000003e-05 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892819+0000", 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003515107 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893354+0000", 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053514000000000003 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893377+0000", 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2265000000000001e-05 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893408+0000", 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1276999999999999e-05 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.244634549000001, 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094233230000000008, 2024-09-16T04:08:18.185 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 162 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884627+0000", 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884629+0000", 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.187e-06 2024-09-16T04:08:18.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884632+0000", 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.501e-06 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884635+0000", 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7389999999999999e-06 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892223+0000", 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075881029999999997 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892291+0000", 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7508000000000005e-05 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892412+0000", 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012124 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893979+0000", 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015676069999999999 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893987+0000", 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.306e-06 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894013+0000", 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6305999999999999e-05 2024-09-16T04:08:18.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894050+0000", 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6492999999999998e-05 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.244622178, 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095013200000000006, 2024-09-16T04:08:18.191 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 163 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2100000000000001e-07 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884642+0000", 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.478e-06 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884646+0000", 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.292e-06 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884648+0000", 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6230000000000001e-06 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892847+0000", 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0081990580000000004 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892891+0000", 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3628999999999999e-05 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892959+0000", 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8151000000000005e-05 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894055+0000", 2024-09-16T04:08:18.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001095927 2024-09-16T04:08:18.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894088+0000", 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3016999999999999e-05 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894117+0000", 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9204999999999999e-05 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894140+0000", 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2719000000000001e-05 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.244608639999999, 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.198 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:18.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:18.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:18.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.244554578999999, 2024-09-16T04:08:18.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:18.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:18.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:18.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:18.210 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 11.020286041, 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:18.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:18.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:18.214 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.214 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.214 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.442352080999999, 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:18.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:18.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:18.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:18.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s2 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.0295778519999992, 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010866819, 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.221 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.222 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:18.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099684+0000", 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.004e-06 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099714+0000", 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9807000000000001e-05 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099769+0000", 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5306000000000002e-05 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099781+0000", 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2091e-05 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099835+0000", 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4058000000000003e-05 2024-09-16T04:08:18.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099845+0000", 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5729999999999997e-06 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110509+0000", 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010664339 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110550+0000", 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640999999999999e-05 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.0255389360000002, 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.226 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:18.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:18.230 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.0241515220000004, 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:18.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:18.234 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.0218844669999996, 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099172860000000009, 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107377+0000", 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8900000000000001e-07 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107437+0000", 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0628000000000003e-05 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107526+0000", 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8873000000000001e-05 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107535+0000", 2024-09-16T04:08:18.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0489999999999993e-06 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111543+0000", 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040080810000000001 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111554+0000", 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0321e-05 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117271+0000", 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057176529999999996 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117294+0000", 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2391999999999998e-05 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 42/27 ECSubWrite(tid=87, reqid=osd.5.0:87, at_version=42'39, trim_to=0'0, roll_forward_to=41'35))", 2024-09-16T04:08:18.239 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.436384+0000", 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.69287657599999997, 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092481160000000007, 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436384+0000", 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:18.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436384+0000", 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436385+0000", 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2799999999999995e-07 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436405+0000", 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0169e-05 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436443+0000", 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7920000000000003e-05 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436451+0000", 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4120000000000001e-06 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436507+0000", 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5512e-05 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436516+0000", 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2699999999999993e-06 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444738+0000", 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082217950000000005 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.445632+0000", 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089431000000000005 2024-09-16T04:08:18.243 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.244 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.244 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.244 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:18.244 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:18.244 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:18.244 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T04:08:18.244 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:08:18.245 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T04:08:18.245 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:08:18.247 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:18.247 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:18.245+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:18.347 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:18.345+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:18.362 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:18.362 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:18.362 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:18.362 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:18.371 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:18.439 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-16T04:08:18.440 INFO:tasks.rados.rados.0.smithi016.stdout:141: done (5 left) 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:142: rmattr oid 38 current snap is 7 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:135: finishing write tid 3 to smithi01639990-26 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:135: finishing write tid 4 to smithi01639990-26 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:135: finishing write tid 5 to smithi01639990-26 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:135: oid 26 updating version 0 to 6 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:135: oid 26 updating version 6 to 8 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:135: oid 26 version 8 is already newer than 7 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:135: oid 26 updating version 8 to 9 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 26 v 9 (ObjNum 80 snap 7 seq_num 80) dirty exists 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:135: left oid 26 (ObjNum 80 snap 7 seq_num 80) 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:135: done (5 left) 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:143: delete oid 3 current snap is 7 2024-09-16T04:08:18.441 INFO:tasks.rados.rados.0.smithi016.stdout:138: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:08:18.449 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:18.447+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:18.457 INFO:tasks.rados.rados.0.smithi016.stdout:137: finishing write tid 1 to smithi01639990-29 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: finishing write tid 2 to smithi01639990-29 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: finishing write tid 3 to smithi01639990-29 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: finishing write tid 4 to smithi01639990-29 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: finishing write tid 5 to smithi01639990-29 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: oid 29 updating version 0 to 31 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: oid 29 version 31 is already newer than 30 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: oid 29 updating version 31 to 32 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: oid 29 updating version 32 to 33 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 29 v 33 (ObjNum 81 snap 7 seq_num 81) dirty exists 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: left oid 29 (ObjNum 81 snap 7 seq_num 81) 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:137: done (5 left) 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:138: done (4 left) 2024-09-16T04:08:18.458 INFO:tasks.rados.rados.0.smithi016.stdout:143: done (3 left) 2024-09-16T04:08:18.459 INFO:tasks.rados.rados.0.smithi016.stdout:144: snap_create 2024-09-16T04:08:18.459 INFO:tasks.rados.rados.0.smithi016.stdout:139: finishing write tid 1 to smithi01639990-46 2024-09-16T04:08:18.459 INFO:tasks.rados.rados.0.smithi016.stdout:139: finishing write tid 2 to smithi01639990-46 2024-09-16T04:08:18.459 INFO:tasks.rados.rados.0.smithi016.stdout:139: finishing write tid 3 to smithi01639990-46 2024-09-16T04:08:18.459 INFO:tasks.rados.rados.0.smithi016.stdout:140: expect (ObjNum 60 snap 2 seq_num 60) 2024-09-16T04:08:18.480 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:18.480 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:18.480 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:18.480 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:18.480 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:18.489 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:18.516 INFO:tasks.rados.rados.0.smithi016.stdout:139: finishing write tid 4 to smithi01639990-46 2024-09-16T04:08:18.516 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T04:08:18.550 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:18.548+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:18.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:08:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:08:18.570 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-09-16T04:08:18.443138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37618227,"num_objects":49,"num_object_clones":1,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":296,"num_read_kb":15385,"num_write":446,"num_write_kb":47948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":146,"acting":146,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":419016,"kb_used_data":73856,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1132043064,"statfs":{"total":1159641169920,"available":1159212097536,"internally_reserved":0,"allocated":75628544,"data_stored":73452338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":353154364},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3522560,"num_objects":-9,"num_object_clones":-6,"num_object_copies":-72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":-2,"num_read":36,"num_read_kb":8012,"num_write":91,"num_write_kb":9952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.770882"},"pg_stats":[{"pgid":"3.f","version":"35'4","reported_seq":57,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.106762+0000","last_change":"2024-09-16T04:08:13.107159+0000","last_active":"2024-09-16T04:08:13.106762+0000","last_peered":"2024-09-16T04:08:13.106762+0000","last_clean":"2024-09-16T04:08:13.106762+0000","last_became_active":"2024-09-16T04:08:03.221547+0000","last_became_peered":"2024-09-16T04:08:03.221547+0000","last_unstale":"2024-09-16T04:08:13.106762+0000","last_undegraded":"2024-09-16T04:08:13.106762+0000","last_fullsized":"2024-09-16T04:08:13.106762+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:22.287430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062754200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":3,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8,2,9,10,0,4],"acting":[7,11,8,2,9,10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.e","version":"34'8","reported_seq":61,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.107821+0000","last_change":"2024-09-16T04:08:13.108889+0000","last_active":"2024-09-16T04:08:13.107821+0000","last_peered":"2024-09-16T04:08:13.107821+0000","last_clean":"2024-09-16T04:08:13.107821+0000","last_became_active":"2024-09-16T04:08:03.220935+0000","last_became_peered":"2024-09-16T04:08:03.220935+0000","last_unstale":"2024-09-16T04:08:13.107821+0000","last_undegraded":"2024-09-16T04:08:13.107821+0000","last_fullsized":"2024-09-16T04:08:13.107821+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:27.232719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024594759999999999,"stat_sum":{"num_bytes":2080768,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":3,"num_write_kb":2512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,1,4,0,9,3,10],"acting":[11,8,1,4,0,9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.d","version":"37'16","reported_seq":96,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.118964+0000","last_change":"2024-09-16T04:08:13.119810+0000","last_active":"2024-09-16T04:08:13.118964+0000","last_peered":"2024-09-16T04:08:13.118964+0000","last_clean":"2024-09-16T04:08:13.118964+0000","last_became_active":"2024-09-16T04:08:07.099169+0000","last_became_peered":"2024-09-16T04:08:07.099169+0000","last_unstale":"2024-09-16T04:08:13.118964+0000","last_undegraded":"2024-09-16T04:08:13.118964+0000","last_fullsized":"2024-09-16T04:08:13.118964+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:18.779993+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.012251714,"stat_sum":{"num_bytes":1818624,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4,11,9,0,1,5],"acting":[7,8,4,11,9,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.c","version":"37'22","reported_seq":88,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.104533+0000","last_change":"2024-09-16T04:08:13.117492+0000","last_active":"2024-09-16T04:08:14.104533+0000","last_peered":"2024-09-16T04:08:14.104533+0000","last_clean":"2024-09-16T04:08:14.104533+0000","last_became_active":"2024-09-16T04:08:03.222541+0000","last_became_peered":"2024-09-16T04:08:03.222541+0000","last_unstale":"2024-09-16T04:08:14.104533+0000","last_undegraded":"2024-09-16T04:08:14.104533+0000","last_fullsized":"2024-09-16T04:08:14.104533+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:24.679380+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.01538963,"stat_sum":{"num_bytes":1458176,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":2647,"num_write":35,"num_write_kb":2896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6,10,9,2,0,11],"acting":[5,3,6,10,9,2,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.b","version":"37'22","reported_seq":79,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.106302+0000","last_change":"2024-09-16T04:08:13.114838+0000","last_active":"2024-09-16T04:08:14.106302+0000","last_peered":"2024-09-16T04:08:14.106302+0000","last_clean":"2024-09-16T04:08:14.106302+0000","last_became_active":"2024-09-16T04:08:03.081648+0000","last_became_peered":"2024-09-16T04:08:03.081648+0000","last_unstale":"2024-09-16T04:08:14.106302+0000","last_undegraded":"2024-09-16T04:08:14.106302+0000","last_fullsized":"2024-09-16T04:08:14.106302+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:06.791740+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.013020238999999999,"stat_sum":{"num_bytes":2048000,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":14,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4,8,11,7,1,6],"acting":[3,0,4,8,11,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.a","version":"35'13","reported_seq":74,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.247475+0000","last_change":"2024-09-16T04:08:13.110366+0000","last_active":"2024-09-16T04:08:14.247475+0000","last_peered":"2024-09-16T04:08:14.247475+0000","last_clean":"2024-09-16T04:08:14.247475+0000","last_became_active":"2024-09-16T04:08:03.219276+0000","last_became_peered":"2024-09-16T04:08:03.219276+0000","last_unstale":"2024-09-16T04:08:14.247475+0000","last_undegraded":"2024-09-16T04:08:14.247475+0000","last_fullsized":"2024-09-16T04:08:14.247475+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:25.127126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032622020000000001,"stat_sum":{"num_bytes":720896,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":7,"num_write_kb":1360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1,10,8,9,3,7],"acting":[6,4,1,10,8,9,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.9","version":"34'29","reported_seq":88,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.102720+0000","last_change":"2024-09-16T04:08:13.103056+0000","last_active":"2024-09-16T04:08:13.102720+0000","last_peered":"2024-09-16T04:08:13.102720+0000","last_clean":"2024-09-16T04:08:13.102720+0000","last_became_active":"2024-09-16T04:08:03.219327+0000","last_became_peered":"2024-09-16T04:08:03.219327+0000","last_unstale":"2024-09-16T04:08:13.102720+0000","last_undegraded":"2024-09-16T04:08:13.102720+0000","last_fullsized":"2024-09-16T04:08:13.102720+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:16.755014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059666600000000004,"stat_sum":{"num_bytes":5292032,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":21,"num_write_kb":5168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,9,5,1,4,10],"acting":[8,11,7,9,5,1,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.8","version":"34'26","reported_seq":78,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.106488+0000","last_change":"2024-09-16T04:08:13.107641+0000","last_active":"2024-09-16T04:08:13.106488+0000","last_peered":"2024-09-16T04:08:13.106488+0000","last_clean":"2024-09-16T04:08:13.106488+0000","last_became_active":"2024-09-16T04:08:03.218478+0000","last_became_peered":"2024-09-16T04:08:03.218478+0000","last_unstale":"2024-09-16T04:08:13.106488+0000","last_undegraded":"2024-09-16T04:08:13.106488+0000","last_fullsized":"2024-09-16T04:08:13.106488+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:19.235019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001320337,"stat_sum":{"num_bytes":2719744,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":13,"num_write_kb":2656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,4,3,5,10,8],"acting":[11,1,2,4,3,5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"1.0","version":"23'137","reported_seq":216,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.106204+0000","last_change":"2024-09-16T04:07:50.119798+0000","last_active":"2024-09-16T04:08:13.106204+0000","last_peered":"2024-09-16T04:08:13.106204+0000","last_clean":"2024-09-16T04:08:13.106204+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:08:13.106204+0000","last_undegraded":"2024-09-16T04:08:13.106204+0000","last_fullsized":"2024-09-16T04:08:13.106204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:07:48.919317+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:53.937788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":50,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.104418+0000","last_change":"2024-09-16T04:07:52.943663+0000","last_active":"2024-09-16T04:08:14.104418+0000","last_peered":"2024-09-16T04:08:14.104418+0000","last_clean":"2024-09-16T04:08:14.104418+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:08:14.104418+0000","last_undegraded":"2024-09-16T04:08:14.104418+0000","last_fullsized":"2024-09-16T04:08:14.104418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.855553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"37'22","reported_seq":94,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.106426+0000","last_change":"2024-09-16T04:08:13.111811+0000","last_active":"2024-09-16T04:08:14.106426+0000","last_peered":"2024-09-16T04:08:14.106426+0000","last_clean":"2024-09-16T04:08:14.106426+0000","last_became_active":"2024-09-16T04:08:03.219487+0000","last_became_peered":"2024-09-16T04:08:03.219487+0000","last_unstale":"2024-09-16T04:08:14.106426+0000","last_undegraded":"2024-09-16T04:08:14.106426+0000","last_fullsized":"2024-09-16T04:08:14.106426+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:31.027468+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.010142490000000001,"stat_sum":{"num_bytes":4653056,"num_objects":6,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":3979,"num_write":35,"num_write_kb":4544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6,4,5,1,8,10],"acting":[3,9,6,4,5,1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.105899+0000","last_change":"2024-09-16T04:07:52.942253+0000","last_active":"2024-09-16T04:08:13.105899+0000","last_peered":"2024-09-16T04:08:13.105899+0000","last_clean":"2024-09-16T04:08:13.105899+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:08:13.105899+0000","last_undegraded":"2024-09-16T04:08:13.105899+0000","last_fullsized":"2024-09-16T04:08:13.105899+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:57.104501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"41'9","reported_seq":80,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:08:16.449087+0000","last_change":"2024-09-16T04:08:16.433714+0000","last_active":"2024-09-16T04:08:16.449087+0000","last_peered":"2024-09-16T04:08:16.449087+0000","last_clean":"2024-09-16T04:08:16.449087+0000","last_became_active":"2024-09-16T04:08:03.221282+0000","last_became_peered":"2024-09-16T04:08:03.221282+0000","last_unstale":"2024-09-16T04:08:16.449087+0000","last_undegraded":"2024-09-16T04:08:16.449087+0000","last_fullsized":"2024-09-16T04:08:16.449087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:02.349584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021073419999999999,"stat_sum":{"num_bytes":2228224,"num_objects":3,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":1300,"num_write":8,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10,11,5,9,6,1],"acting":[0,4,10,11,5,9,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.233015+0000","last_change":"2024-09-16T04:07:53.018517+0000","last_active":"2024-09-16T04:08:14.233015+0000","last_peered":"2024-09-16T04:08:14.233015+0000","last_clean":"2024-09-16T04:08:14.233015+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:08:14.233015+0000","last_undegraded":"2024-09-16T04:08:14.233015+0000","last_fullsized":"2024-09-16T04:08:14.233015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"30'2","reported_seq":55,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.106293+0000","last_change":"2024-09-16T04:08:13.107882+0000","last_active":"2024-09-16T04:08:13.106293+0000","last_peered":"2024-09-16T04:08:13.106293+0000","last_clean":"2024-09-16T04:08:13.106293+0000","last_became_active":"2024-09-16T04:08:03.208071+0000","last_became_peered":"2024-09-16T04:08:03.208071+0000","last_unstale":"2024-09-16T04:08:13.106293+0000","last_undegraded":"2024-09-16T04:08:13.106293+0000","last_fullsized":"2024-09-16T04:08:13.106293+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:09.884377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017772440000000001,"stat_sum":{"num_bytes":196608,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6,1,7,8,0,5],"acting":[11,9,6,1,7,8,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.2","version":"23'2","reported_seq":51,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.104330+0000","last_change":"2024-09-16T04:07:52.944002+0000","last_active":"2024-09-16T04:08:14.104330+0000","last_peered":"2024-09-16T04:08:14.104330+0000","last_clean":"2024-09-16T04:08:14.104330+0000","last_became_active":"2024-09-16T04:07:50.935310+0000","last_became_peered":"2024-09-16T04:07:50.935310+0000","last_unstale":"2024-09-16T04:08:14.104330+0000","last_undegraded":"2024-09-16T04:08:14.104330+0000","last_fullsized":"2024-09-16T04:08:14.104330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:03.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"37'16","reported_seq":79,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.103787+0000","last_change":"2024-09-16T04:08:13.117425+0000","last_active":"2024-09-16T04:08:14.103787+0000","last_peered":"2024-09-16T04:08:14.103787+0000","last_clean":"2024-09-16T04:08:14.103787+0000","last_became_active":"2024-09-16T04:08:03.218410+0000","last_became_peered":"2024-09-16T04:08:03.218410+0000","last_unstale":"2024-09-16T04:08:14.103787+0000","last_undegraded":"2024-09-16T04:08:14.103787+0000","last_fullsized":"2024-09-16T04:08:14.103787+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:29.587549+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012151321,"stat_sum":{"num_bytes":1048576,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":1156,"num_write":8,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,6,11,5,9,10],"acting":[4,3,0,6,11,5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"3.5","version":"37'24","reported_seq":85,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.104575+0000","last_change":"2024-09-16T04:08:13.115886+0000","last_active":"2024-09-16T04:08:14.104575+0000","last_peered":"2024-09-16T04:08:14.104575+0000","last_clean":"2024-09-16T04:08:14.104575+0000","last_became_active":"2024-09-16T04:08:03.221297+0000","last_became_peered":"2024-09-16T04:08:03.221297+0000","last_unstale":"2024-09-16T04:08:14.104575+0000","last_undegraded":"2024-09-16T04:08:14.104575+0000","last_fullsized":"2024-09-16T04:08:14.104575+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:04.296150+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013875126,"stat_sum":{"num_bytes":4210688,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":16,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6,7,1,11,10],"acting":[5,3,2,6,7,1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.109513+0000","last_change":"2024-09-16T04:07:52.946320+0000","last_active":"2024-09-16T04:08:13.109513+0000","last_peered":"2024-09-16T04:08:13.109513+0000","last_clean":"2024-09-16T04:08:13.109513+0000","last_became_active":"2024-09-16T04:07:50.936587+0000","last_became_peered":"2024-09-16T04:07:50.936587+0000","last_unstale":"2024-09-16T04:08:13.109513+0000","last_undegraded":"2024-09-16T04:08:13.109513+0000","last_fullsized":"2024-09-16T04:08:13.109513+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:13.247770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051151899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"37'17","reported_seq":79,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.121336+0000","last_change":"2024-09-16T04:08:13.122332+0000","last_active":"2024-09-16T04:08:13.121336+0000","last_peered":"2024-09-16T04:08:13.121336+0000","last_clean":"2024-09-16T04:08:13.121336+0000","last_became_active":"2024-09-16T04:08:03.219952+0000","last_became_peered":"2024-09-16T04:08:03.219952+0000","last_unstale":"2024-09-16T04:08:13.121336+0000","last_undegraded":"2024-09-16T04:08:13.121336+0000","last_fullsized":"2024-09-16T04:08:13.121336+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:13.772152+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.012658922,"stat_sum":{"num_bytes":638976,"num_objects":3,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":2214,"num_write":30,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,9,5,10,2,6,0],"acting":[1,11,9,5,10,2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":46,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.102611+0000","last_change":"2024-09-16T04:07:52.939889+0000","last_active":"2024-09-16T04:08:13.102611+0000","last_peered":"2024-09-16T04:08:13.102611+0000","last_clean":"2024-09-16T04:08:13.102611+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:08:13.102611+0000","last_undegraded":"2024-09-16T04:08:13.102611+0000","last_fullsized":"2024-09-16T04:08:13.102611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:16.774685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"37'43","reported_seq":108,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.106345+0000","last_change":"2024-09-16T04:08:13.122202+0000","last_active":"2024-09-16T04:08:14.106345+0000","last_peered":"2024-09-16T04:08:14.106345+0000","last_clean":"2024-09-16T04:08:14.106345+0000","last_became_active":"2024-09-16T04:08:03.223216+0000","last_became_peered":"2024-09-16T04:08:03.223216+0000","last_unstale":"2024-09-16T04:08:14.106345+0000","last_undegraded":"2024-09-16T04:08:14.106345+0000","last_fullsized":"2024-09-16T04:08:14.106345+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:26.965102+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.020027028999999998,"stat_sum":{"num_bytes":5439488,"num_objects":4,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":3945,"num_write":47,"num_write_kb":8128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7,0,10,9,4,1],"acting":[3,11,7,0,10,9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":47,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T04:08:13.110193+0000","last_change":"2024-09-16T04:07:52.946321+0000","last_active":"2024-09-16T04:08:13.110193+0000","last_peered":"2024-09-16T04:08:13.110193+0000","last_clean":"2024-09-16T04:08:13.110193+0000","last_became_active":"2024-09-16T04:07:50.937245+0000","last_became_peered":"2024-09-16T04:07:50.937245+0000","last_unstale":"2024-09-16T04:08:13.110193+0000","last_undegraded":"2024-09-16T04:08:13.110193+0000","last_fullsized":"2024-09-16T04:08:13.110193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:18.129176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051620099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"39'17","reported_seq":86,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T04:08:16.433350+0000","last_change":"2024-09-16T04:08:16.433799+0000","last_active":"2024-09-16T04:08:16.433350+0000","last_peered":"2024-09-16T04:08:16.433350+0000","last_clean":"2024-09-16T04:08:16.433350+0000","last_became_active":"2024-09-16T04:08:03.220520+0000","last_became_peered":"2024-09-16T04:08:03.220520+0000","last_unstale":"2024-09-16T04:08:16.433350+0000","last_undegraded":"2024-09-16T04:08:16.433350+0000","last_fullsized":"2024-09-16T04:08:16.433350+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:31.419285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019088899999999999,"stat_sum":{"num_bytes":2473984,"num_objects":2,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":12,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7,8,10,9,5],"acting":[0,1,4,7,8,10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"3"}]},{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T04:08:14.247484+0000","last_change":"2024-09-16T04:07:52.941363+0000","last_active":"2024-09-16T04:08:14.247484+0000","last_peered":"2024-09-16T04:08:14.247484+0000","last_clean":"2024-09-16T04:08:14.247484+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:08:14.247484+0000","last_undegraded":"2024-09-16T04:08:14.247484+0000","last_fullsized":"2024-09-16T04:08:14.247484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:08:59.932581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":37027840,"num_objects":45,"num_object_clones":1,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":170,"num_read_kb":15276,"num_write":263,"num_write_kb":45008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69763072,"data_stored":69931008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":128,"acting":128,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378632,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31284,"kb_used_data":4252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340556,"statfs":{"total":96636764160,"available":96604729344,"internally_reserved":0,"allocated":4354048,"data_stored":4161914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378631,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37476,"kb_used_data":6284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334364,"statfs":{"total":96636764160,"available":96598388736,"internally_reserved":0,"allocated":6434816,"data_stored":6234707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378631,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34220,"kb_used_data":7192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337620,"statfs":{"total":96636764160,"available":96601722880,"internally_reserved":0,"allocated":7364608,"data_stored":7172474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378634,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37524,"kb_used_data":6336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334316,"statfs":{"total":96636764160,"available":96598339584,"internally_reserved":0,"allocated":6488064,"data_stored":6295930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378632,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33788,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338052,"statfs":{"total":96636764160,"available":96602165248,"internally_reserved":0,"allocated":2662400,"data_stored":2466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411335,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38884,"kb_used_data":7696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332956,"statfs":{"total":96636764160,"available":96596946944,"internally_reserved":0,"allocated":7880704,"data_stored":7685018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411337,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34816,"kb_used_data":7788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337024,"statfs":{"total":96636764160,"available":96601112576,"internally_reserved":0,"allocated":7974912,"data_stored":7782778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411335,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33616,"kb_used_data":6588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338224,"statfs":{"total":96636764160,"available":96602341376,"internally_reserved":0,"allocated":6746112,"data_stored":6553978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411335,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":5444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":5574656,"data_stored":5382522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411337,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":6012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":6156288,"data_stored":5960602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411336,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35204,"kb_used_data":8172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336636,"statfs":{"total":96636764160,"available":96600715264,"internally_reserved":0,"allocated":8368128,"data_stored":8171917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444041,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":5492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":5623808,"data_stored":5584130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5181440,"data_stored":5349376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7970816,"data_stored":7970816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2256896,"data_stored":2256896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6094848,"data_stored":6094848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6971392,"data_stored":6971392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6025216,"data_stored":6025216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3960832,"data_stored":3960832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5230592,"data_stored":5230592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5181440,"data_stored":5181440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":6352896,"data_stored":6352896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":7581696,"data_stored":7581696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6955008,"data_stored":6955008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:08:18.571 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-16T04:08:18.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:186 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~589824] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.194343+0000", 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.396576996, 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060370950000000001, 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.594 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 186 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.194343+0000", 2024-09-16T04:08:18.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.194343+0000", 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.194345+0000", 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.759e-06 2024-09-16T04:08:18.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195173+0000", 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00082836499999999996 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195190+0000", 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6756999999999999e-05 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195198+0000", 2024-09-16T04:08:18.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2599999999999999e-06 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195250+0000", 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1996999999999997e-05 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195323+0000", 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3161e-05 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196249+0000", 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00092679299999999997 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200145+0000", 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038957269999999999 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200279+0000", 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013368599999999999 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200380+0000", 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010158999999999999 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:187 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~376832] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.196468+0000", 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.394452297999999, 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053497429999999997, 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.600 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 187 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196468+0000", 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196468+0000", 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196469+0000", 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.299e-06 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197470+0000", 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010008770000000001 2024-09-16T04:08:18.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197486+0000", 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6186000000000001e-05 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197496+0000", 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.268e-06 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197604+0000", 2024-09-16T04:08:18.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108723 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197700+0000", 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5773999999999995e-05 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.198516+0000", 2024-09-16T04:08:18.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081628899999999999 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200458+0000", 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001941826 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201745+0000", 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001287049 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.201818+0000", 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2452000000000004e-05 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:189 3.9s0 3:92dcccbe:::smithi01639990-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.197539+0000", 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.393381247000001, 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005956355, 2024-09-16T04:08:18.606 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 189 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197539+0000", 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197539+0000", 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197541+0000", 2024-09-16T04:08:18.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0770000000000002e-06 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197543+0000", 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1859999999999999e-06 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197550+0000", 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7020000000000004e-06 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197555+0000", 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7809999999999996e-06 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200575+0000", 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030196649999999999 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200666+0000", 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1508999999999994e-05 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202294+0000", 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016276719999999999 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:18.611 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202368+0000", 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4441000000000001e-05 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.203495+0000", 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001127322 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(osd.11.18:1 3.9s0 3:92dcccbe:::smithi01639990-16:head [assert-version v23,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.214792+0000", 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.376128331, 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063663399999999998, 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214792+0000", 2024-09-16T04:08:18.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214792+0000", 2024-09-16T04:08:18.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214794+0000", 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9640000000000002e-06 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214802+0000", 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9699999999999999e-06 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214822+0000", 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9737000000000001e-05 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214828+0000", 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0159999999999997e-06 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.616 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214901+0000", 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3320999999999998e-05 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214983+0000", 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2216e-05 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.221158+0000", 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061751159999999996 2024-09-16T04:08:18.617 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.9s0 32/27 ECSubRead(tid=36, to_read={3:92dcccbe:::smithi01639990-16:head=0,520192,0}, subchunks={3:92dcccbe:::smithi01639990-16:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.375716206, 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053324849999999997, 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:18.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215204+0000", 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215226+0000", 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1671e-05 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215235+0000", 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.329e-06 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.215280+0000", 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5154000000000002e-05 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.220537+0000", 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0052563310000000004 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.621 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.4874268879999999, 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073296819999999997, 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.622 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103494+0000", 2024-09-16T04:08:18.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1399999999999997e-07 2024-09-16T04:08:18.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103514+0000", 2024-09-16T04:08:18.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9698999999999999e-05 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103592+0000", 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8378000000000001e-05 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103598+0000", 2024-09-16T04:08:18.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9100000000000002e-06 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105790+0000", 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021916359999999998 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105798+0000", 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2490000000000001e-06 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110788+0000", 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049902660000000001 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110823+0000", 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5129999999999997e-05 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=34, reqid=client.4440.0:211, at_version=33'20, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.4752099520000002, 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078650630000000003, 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115711+0000", 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2399999999999998e-07 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115771+0000", 2024-09-16T04:08:18.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9478999999999998e-05 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119749+0000", 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039784099999999999 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119760+0000", 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0664e-05 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119899+0000", 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013905300000000001 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119919+0000", 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9854999999999999e-05 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123544+0000", 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036256499999999998 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123575+0000", 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1328000000000002e-05 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=35, reqid=client.4440.0:212, at_version=33'21, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.4724548300000002, 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077855709999999998, 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.632 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118551+0000", 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4878999999999998e-05 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120595+0000", 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020432639999999999 2024-09-16T04:08:18.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120604+0000", 2024-09-16T04:08:18.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0250000000000008e-06 2024-09-16T04:08:18.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122166+0000", 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015625599999999999 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122175+0000", 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8470000000000002e-06 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126217+0000", 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040415249999999998 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126251+0000", 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4570999999999998e-05 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.4492270670000007, 2024-09-16T04:08:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:18.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 37/27 ECSubWrite(tid=128, reqid=osd.3.0:124, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:18.643 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111259+0000", 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.4796613079999998, 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054317890000000002, 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111259+0000", 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111259+0000", 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111260+0000", 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5899999999999996e-07 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111269+0000", 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0950000000000002e-06 2024-09-16T04:08:18.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111299+0000", 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0320000000000001e-05 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111308+0000", 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7430000000000008e-06 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111360+0000", 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1959000000000002e-05 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111369+0000", 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0829999999999993e-06 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116129+0000", 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047604600000000002 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116691+0000", 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00056147 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.648 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:296 3.9s0 3:9fd10cc9:::smithi01639990-50:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.4863471769999999, 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010626313, 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.649 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:18.650 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:18.650 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:18.650 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296 2024-09-16T04:08:18.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.650 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:18.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104575+0000", 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1349999999999999e-06 2024-09-16T04:08:18.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104586+0000", 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0603e-05 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104606+0000", 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9840999999999999e-05 2024-09-16T04:08:18.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104615+0000", 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0189999999999995e-06 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104674+0000", 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9534999999999998e-05 2024-09-16T04:08:18.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104680+0000", 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9320000000000001e-06 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109134+0000", 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044534120000000003 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109195+0000", 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0983000000000002e-05 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109711+0000", 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051621600000000001 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.110753+0000", 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001042533 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115115+0000", 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043613030000000004 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115200+0000", 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4801000000000003e-05 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.656 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 42/27 ECSubWrite(tid=154, reqid=osd.3.0:154, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439636+0000", 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.151284704, 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062793340000000001, 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439636+0000", 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439636+0000", 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439636+0000", 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8599999999999999e-07 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439650+0000", 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3954000000000001e-05 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440315+0000", 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066470199999999998 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440324+0000", 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5789999999999993e-06 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440849+0000", 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052441 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440859+0000", 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0713000000000001e-05 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.445872+0000", 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050127039999999998 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.445915+0000", 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2985999999999998e-05 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 42/27 ECSubWrite(tid=50, reqid=osd.11.0:50, at_version=42'32, trim_to=0'0, roll_forward_to=41'30))", 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.442299+0000", 2024-09-16T04:08:18.661 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.148621535, 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050765369999999999, 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442299+0000", 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442299+0000", 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442299+0000", 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5499999999999999e-07 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442326+0000", 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6999999999999999e-05 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442374+0000", 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8016999999999998e-05 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442383+0000", 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1530000000000003e-06 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442449+0000", 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5413000000000005e-05 2024-09-16T04:08:18.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442459+0000", 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0193e-05 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447337+0000", 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048777719999999998 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447375+0000", 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8733999999999999e-05 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443622+0000", 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.1472983999999999, 2024-09-16T04:08:18.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0071718909999999997, 2024-09-16T04:08:18.667 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.667 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.667 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.667 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443622+0000", 2024-09-16T04:08:18.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443622+0000", 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443623+0000", 2024-09-16T04:08:18.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.037e-06 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443939+0000", 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00031647100000000003 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447492+0000", 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035522990000000001 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447506+0000", 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4268e-05 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447577+0000", 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0769e-05 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447587+0000", 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0406e-05 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450760+0000", 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031730220000000002 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450794+0000", 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3618999999999998e-05 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.141897806, 2024-09-16T04:08:18.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085197069999999996, 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:18.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4000000000000001e-07 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449160+0000", 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013637999999999999 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451237+0000", 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020768179999999998 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451246+0000", 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451532+0000", 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00028571800000000002 2024-09-16T04:08:18.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451542+0000", 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0118999999999999e-05 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457500+0000", 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059576680000000002 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457542+0000", 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2716999999999998e-05 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=160, reqid=client.4440.0:347, at_version=42'32, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.451166+0000", 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.1397544769999999, 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064616450000000002, 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.677 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451166+0000", 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451166+0000", 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451167+0000", 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0809999999999999e-06 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451210+0000", 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3350000000000003e-05 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452031+0000", 2024-09-16T04:08:18.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00082036499999999998 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452040+0000", 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7890000000000006e-06 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452121+0000", 2024-09-16T04:08:18.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0725999999999999e-05 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452148+0000", 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6863999999999999e-05 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457572+0000", 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054240030000000002 2024-09-16T04:08:18.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457628+0000", 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5467000000000003e-05 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=161, reqid=client.4440.0:348, at_version=42'33, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:18.682 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.451985+0000", 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.1389355110000001, 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056726499999999996, 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451985+0000", 2024-09-16T04:08:18.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451985+0000", 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451986+0000", 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8899999999999998e-07 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452006+0000", 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0486999999999999e-05 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452781+0000", 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00077455899999999997 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452790+0000", 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.397e-06 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452896+0000", 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106106 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452906+0000", 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4069999999999999e-06 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457631+0000", 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047248689999999996 2024-09-16T04:08:18.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457658+0000", 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6835999999999999e-05 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.1311714589999999, 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:08:18.687 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:08:18.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:08:18.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0006292999999999, 2024-09-16T04:08:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:08:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:08:18.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:363 3.9s0 3:94de6fe1:::smithi01639990-3:head [stat,delete] snapc 8=[6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:08:18.696 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.15106264, 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085012569999999999, 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 363 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439860+0000", 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439869+0000", 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3170000000000008e-06 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439888+0000", 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9293e-05 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439897+0000", 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:18.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439953+0000", 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6317999999999998e-05 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439959+0000", 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3040000000000004e-06 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:18.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441596+0000", 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001637655 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441659+0000", 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2658999999999998e-05 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.442297+0000", 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063811800000000002 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.443562+0000", 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001265361 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448304+0000", 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047413560000000004 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448359+0000", 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5189000000000003e-05 2024-09-16T04:08:18.702 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.703 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.703 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.703 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:18.703 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:18.703 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:18.704 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:18.702+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:18.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:18.805 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:18.803+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:18.808 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:18.808 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:18.808 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:18.808 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:18.817 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:18.889 INFO:tasks.rados.rados.0.smithi016.stdout:139: finishing write tid 5 to smithi01639990-46 2024-09-16T04:08:18.889 INFO:tasks.rados.rados.0.smithi016.stdout:139: oid 46 updating version 0 to 37 2024-09-16T04:08:18.889 INFO:tasks.rados.rados.0.smithi016.stdout:139: oid 46 updating version 37 to 38 2024-09-16T04:08:18.889 INFO:tasks.rados.rados.0.smithi016.stdout:139: oid 46 version 38 is already newer than 35 2024-09-16T04:08:18.889 INFO:tasks.rados.rados.0.smithi016.stdout:139: oid 46 version 38 is already newer than 36 2024-09-16T04:08:18.890 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 46 v 38 (ObjNum 82 snap 7 seq_num 82) dirty exists 2024-09-16T04:08:18.890 INFO:tasks.rados.rados.0.smithi016.stdout:139: left oid 46 (ObjNum 82 snap 7 seq_num 82) 2024-09-16T04:08:18.906 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:18.903+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:18.921 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:18.921 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:18.921 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:18.921 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:18.922 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:18.930 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:19.007 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:19.005+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:19.034 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:19.034 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:19.034 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.937711051000001, 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.035 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.037 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:19.038 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:19.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.039 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.039 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:19.039 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:19.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.039 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es5 32/27 ECSubWrite(tid=28, reqid=client.4440.0:197, at_version=32'4, trim_to=0'0, roll_forward_to=30'2))", 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.227608+0000", 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.804410493000001, 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004862318, 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.040 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227608+0000", 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227608+0000", 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227610+0000", 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8369999999999999e-06 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228034+0000", 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042372999999999998 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228094+0000", 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0418000000000003e-05 2024-09-16T04:08:19.042 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228107+0000", 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3121e-05 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228200+0000", 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2559000000000006e-05 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228213+0000", 2024-09-16T04:08:19.043 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3574000000000001e-05 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232430+0000", 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004217213 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232470+0000", 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9866000000000001e-05 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.044 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as5 33/27 ECSubWrite(tid=14, reqid=osd.6.0:14, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.095198+0000", 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.9368200889999994, 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044375830000000002, 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095198+0000", 2024-09-16T04:08:19.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095198+0000", 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095199+0000", 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2999999999999999e-07 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095220+0000", 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1407999999999999e-05 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095852+0000", 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063245300000000005 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095862+0000", 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0550000000000005e-06 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096043+0000", 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018101900000000001 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096052+0000", 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7909999999999999e-06 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099602+0000", 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035497279999999998 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099636+0000", 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3899000000000002e-05 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 33/27 ECSubWrite(tid=37, reqid=osd.8.0:37, at_version=33'25, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.098805+0000", 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.9332130710000008, 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043258790000000004, 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.049 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098805+0000", 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098805+0000", 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098806+0000", 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.23e-07 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098827+0000", 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1308e-05 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098879+0000", 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1863000000000001e-05 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098886+0000", 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6669999999999996e-06 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098947+0000", 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0789999999999999e-05 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.098956+0000", 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.49e-06 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103100+0000", 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041444289999999998 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103131+0000", 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1109000000000003e-05 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.053 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.054 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.054 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:19.054 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:19.054 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.9322705780000007, 2024-09-16T04:08:19.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067064200000000003, 2024-09-16T04:08:19.054 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099749+0000", 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1299999999999999e-07 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099771+0000", 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2642000000000001e-05 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099811+0000", 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9406999999999997e-05 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099819+0000", 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.057 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100401+0000", 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058156199999999999 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100411+0000", 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7079999999999992e-06 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106408+0000", 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059972369999999999 2024-09-16T04:08:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106454+0000", 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6384000000000002e-05 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es5 33/27 ECSubWrite(tid=31, reqid=osd.11.0:31, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.101895+0000", 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.9301236599999996, 2024-09-16T04:08:19.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046241909999999997, 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101895+0000", 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101895+0000", 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101895+0000", 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.73e-07 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.101941+0000", 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5605999999999997e-05 2024-09-16T04:08:19.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102038+0000", 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7359000000000001e-05 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102048+0000", 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5740000000000002e-06 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102115+0000", 2024-09-16T04:08:19.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7688000000000007e-05 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102125+0000", 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4569999999999994e-06 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106479+0000", 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043546799999999997 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106519+0000", 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9653999999999997e-05 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.102002+0000", 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.9300168430000006, 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045426470000000004, 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.064 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102002+0000", 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102002+0000", 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102002+0000", 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9299999999999999e-07 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102018+0000", 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6099999999999998e-05 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102101+0000", 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2735e-05 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.067 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102108+0000", 2024-09-16T04:08:19.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7230000000000004e-06 2024-09-16T04:08:19.067 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.067 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102214+0000", 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105642 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102223+0000", 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.149e-06 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106523+0000", 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0042998619999999998 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106544+0000", 2024-09-16T04:08:19.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1243e-05 2024-09-16T04:08:19.069 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.069 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.069 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.069 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:19.070 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103167+0000", 2024-09-16T04:08:19.070 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.9288515119999996, 2024-09-16T04:08:19.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0055620829999999998, 2024-09-16T04:08:19.070 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.070 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.070 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.070 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103167+0000", 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103167+0000", 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103175+0000", 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2879999999999993e-06 2024-09-16T04:08:19.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.072 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103717+0000", 2024-09-16T04:08:19.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00054184999999999997 2024-09-16T04:08:19.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103749+0000", 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2314999999999998e-05 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103757+0000", 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2729999999999997e-06 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103808+0000", 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1072999999999997e-05 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103817+0000", 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8410000000000006e-06 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108704+0000", 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048869250000000003 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108729+0000", 2024-09-16T04:08:19.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5517999999999999e-05 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103841+0000", 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.9281773910000002, 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048534260000000001, 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.075 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103841+0000", 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103841+0000", 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103842+0000", 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4700000000000001e-07 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103856+0000", 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4748e-05 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103886+0000", 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9437999999999998e-05 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103894+0000", 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9189999999999999e-06 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103978+0000", 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3775000000000003e-05 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103983+0000", 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.519e-06 2024-09-16T04:08:19.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108661+0000", 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004677858 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108694+0000", 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3522000000000002e-05 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.9278236660000001, 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062253660000000004, 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104196+0000", 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104218+0000", 2024-09-16T04:08:19.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1974999999999999e-05 2024-09-16T04:08:19.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104245+0000", 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7489999999999999e-05 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104254+0000", 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104313+0000", 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9151000000000003e-05 2024-09-16T04:08:19.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104322+0000", 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110389+0000", 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006067205 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110420+0000", 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1100999999999999e-05 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.106795+0000", 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.92522299, 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056249589999999997, 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.085 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106795+0000", 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106795+0000", 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106796+0000", 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.79e-07 2024-09-16T04:08:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106819+0000", 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3085999999999999e-05 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106897+0000", 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8067999999999999e-05 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106905+0000", 2024-09-16T04:08:19.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2209999999999997e-06 2024-09-16T04:08:19.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106978+0000", 2024-09-16T04:08:19.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2822000000000002e-05 2024-09-16T04:08:19.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106988+0000", 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0468999999999999e-05 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112382+0000", 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005393424 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112420+0000", 2024-09-16T04:08:19.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8689999999999997e-05 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=163, reqid=osd.7.0:163, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.92204438, 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053772749999999999, 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.090 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1799999999999999e-07 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110009+0000", 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4467000000000001e-05 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110085+0000", 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6359999999999994e-05 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110093+0000", 2024-09-16T04:08:19.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0879999999999995e-06 2024-09-16T04:08:19.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110482+0000", 2024-09-16T04:08:19.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00038852100000000002 2024-09-16T04:08:19.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110498+0000", 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5858999999999998e-05 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115317+0000", 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048192579999999999 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115351+0000", 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4504e-05 2024-09-16T04:08:19.094 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.9220071440000002, 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062070839999999999, 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:19.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:19.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110012+0000", 2024-09-16T04:08:19.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3999999999999997e-07 2024-09-16T04:08:19.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110025+0000", 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3053e-05 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110784+0000", 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075944100000000002 2024-09-16T04:08:19.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110793+0000", 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4880000000000007e-06 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110853+0000", 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0333e-05 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110861+0000", 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2090000000000005e-06 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116184+0000", 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053227389999999999 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116218+0000", 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4480999999999997e-05 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 37/27 ECSubWrite(tid=128, reqid=osd.3.0:124, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111261+0000", 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.9207571799999998, 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0041541820000000002, 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111261+0000", 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111261+0000", 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111262+0000", 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8100000000000002e-07 2024-09-16T04:08:19.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111273+0000", 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0592e-05 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111299+0000", 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6585999999999999e-05 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111308+0000", 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7350000000000002e-06 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111384+0000", 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5816999999999996e-05 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111393+0000", 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2799999999999992e-06 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115386+0000", 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039930820000000002 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115415+0000", 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9408999999999999e-05 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.105 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.4s2 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111488+0000", 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.9205308560000001, 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047796619999999996, 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111488+0000", 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111488+0000", 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111488+0000", 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8199999999999999e-07 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111508+0000", 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0550000000000001e-05 2024-09-16T04:08:19.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111540+0000", 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1263000000000002e-05 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111548+0000", 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7690000000000002e-06 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111604+0000", 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5924999999999997e-05 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111613+0000", 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5050000000000007e-06 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116241+0000", 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046280510000000002 2024-09-16T04:08:19.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116267+0000", 2024-09-16T04:08:19.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6417000000000001e-05 2024-09-16T04:08:19.110 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.110 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.110 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 37/27 ECSubWrite(tid=129, reqid=osd.3.0:129, at_version=37'43, trim_to=0'0, roll_forward_to=35'41))", 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.9192413579999998, 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061976449999999999, 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112778+0000", 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3499999999999996e-07 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112797+0000", 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9578000000000001e-05 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112842+0000", 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5093999999999999e-05 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112850+0000", 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112972+0000", 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000122033 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112982+0000", 2024-09-16T04:08:19.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5789999999999993e-06 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118944+0000", 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005961816 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118975+0000", 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0933999999999999e-05 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.115 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=155, reqid=osd.3.0:155, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.5923782449999999, 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073085160000000001, 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:19.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1299999999999999e-07 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439663+0000", 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3030000000000001e-05 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439705+0000", 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1078999999999998e-05 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439714+0000", 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441461+0000", 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017466350000000001 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441475+0000", 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4404e-05 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446911+0000", 2024-09-16T04:08:19.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054361430000000001 2024-09-16T04:08:19.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446949+0000", 2024-09-16T04:08:19.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7465000000000001e-05 2024-09-16T04:08:19.120 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.5882994780000002, 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068399539999999997, 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.121 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443720+0000", 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9500000000000001e-07 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444146+0000", 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042652899999999998 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444178+0000", 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1412999999999999e-05 2024-09-16T04:08:19.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444185+0000", 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4270000000000002e-06 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444247+0000", 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1760000000000005e-05 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444254+0000", 2024-09-16T04:08:19.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9260000000000003e-06 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450519+0000", 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062649960000000001 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450559+0000", 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0108000000000003e-05 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.125 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s5 42/27 ECSubWrite(tid=40, reqid=osd.0.0:40, at_version=42'11, trim_to=0'0, roll_forward_to=41'9))", 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.58812547, 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067397389999999998, 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.126 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0700000000000001e-07 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443914+0000", 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1274e-05 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443937+0000", 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2991999999999999e-05 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443945+0000", 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4780000000000002e-06 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444021+0000", 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5814000000000003e-05 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444033+0000", 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2322e-05 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450586+0000", 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065529869999999997 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450633+0000", 2024-09-16T04:08:19.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6764999999999997e-05 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 43/27 ECSubWrite(tid=51, reqid=client.4440.0:363, at_version=0'0, trim_to=0'0, roll_forward_to=43'35))", 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.449574+0000", 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.58244461300000006, 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044676840000000004, 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.449574+0000", 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.449574+0000", 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.449575+0000", 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5499999999999997e-07 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.449586+0000", 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1589e-05 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.449605+0000", 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.878e-05 2024-09-16T04:08:19.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.449614+0000", 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8119999999999997e-06 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.449677+0000", 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3448000000000006e-05 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.449687+0000", 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5040000000000008e-06 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.454005+0000", 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043182660000000003 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.454042+0000", 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6529999999999998e-05 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.135 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.136 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.136 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.136 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.136 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:19.137 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:19.138 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:19.134+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:19.240 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:19.235+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:19.284 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:19.284 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:19.284 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:19.284 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:19.292 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:19.339 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:19.337+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:19.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:19.396 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:19.396 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:19.396 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:19.396 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:19.396 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:19.404 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:19.440 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:19.437+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:19.444 INFO:teuthology.orchestra.run.smithi016.stderr:set pool 3 pg_num to 26 2024-09-16T04:08:19.445 INFO:tasks.rados.rados.0.smithi016.stdout:139: done (3 left) 2024-09-16T04:08:19.445 INFO:tasks.rados.rados.0.smithi016.stdout:140: done (2 left) 2024-09-16T04:08:19.445 INFO:tasks.rados.rados.0.smithi016.stdout:142: done (1 left) 2024-09-16T04:08:19.445 INFO:tasks.rados.rados.0.smithi016.stdout:144: done (0 left) 2024-09-16T04:08:19.445 INFO:tasks.rados.rados.0.smithi016.stdout:145: rollback oid 46 current snap is 8 2024-09-16T04:08:19.445 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 46 to 7 2024-09-16T04:08:19.445 INFO:tasks.rados.rados.0.smithi016.stdout:146: read oid 5 snap -1 2024-09-16T04:08:19.446 INFO:tasks.rados.rados.0.smithi016.stdout:146: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:08:19.446 INFO:tasks.rados.rados.0.smithi016.stdout:147: snap_remove snap 4 2024-09-16T04:08:19.494 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:19.494 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:19.511 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:19.511 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:19.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:19.511 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:19.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.512 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es7 32/27 ECSubWrite(tid=28, reqid=client.4440.0:197, at_version=32'4, trim_to=0'0, roll_forward_to=30'2))", 2024-09-16T04:08:19.512 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.227987+0000", 2024-09-16T04:08:19.512 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.280777322, 2024-09-16T04:08:19.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048430519999999996, 2024-09-16T04:08:19.512 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.512 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.513 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.513 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.513 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.513 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.513 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227987+0000", 2024-09-16T04:08:19.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.513 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227987+0000", 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227989+0000", 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3650000000000001e-06 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228349+0000", 2024-09-16T04:08:19.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036041099999999999 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228398+0000", 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9086999999999999e-05 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228408+0000", 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4420000000000004e-06 2024-09-16T04:08:19.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228489+0000", 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1699000000000005e-05 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228500+0000", 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0516000000000001e-05 2024-09-16T04:08:19.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232804+0000", 2024-09-16T04:08:19.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043043430000000004 2024-09-16T04:08:19.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232830+0000", 2024-09-16T04:08:19.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6188999999999999e-05 2024-09-16T04:08:19.518 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.518 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.518 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.518 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 33/27 ECSubWrite(tid=77, reqid=osd.3.0:77, at_version=33'19, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:19.518 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.095188+0000", 2024-09-16T04:08:19.518 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.413576492000001, 2024-09-16T04:08:19.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050055009999999999, 2024-09-16T04:08:19.519 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.519 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.519 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.519 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095188+0000", 2024-09-16T04:08:19.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095188+0000", 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095189+0000", 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3499999999999996e-07 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.520 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.247e-05 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095237+0000", 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5673999999999998e-05 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095246+0000", 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0170000000000002e-06 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095304+0000", 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8627e-05 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095316+0000", 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1095e-05 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100160+0000", 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048447230000000004 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100194+0000", 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3259999999999997e-05 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as3 33/27 ECSubWrite(tid=14, reqid=osd.6.0:14, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.413553636, 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004947758, 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.523 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.524 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.524 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:19.524 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.524 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095211+0000", 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0900000000000001e-07 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095231+0000", 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9673e-05 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095274+0000", 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3550000000000001e-05 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095283+0000", 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8740000000000001e-06 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095358+0000", 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4351999999999996e-05 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.526 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.095366+0000", 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9160000000000007e-06 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100124+0000", 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047573499999999996 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100159+0000", 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4833999999999999e-05 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.527 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s7 33/27 ECSubWrite(tid=13, reqid=osd.4.0:13, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.096050+0000", 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.412714622999999, 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005993473, 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.528 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096050+0000", 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096050+0000", 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096050+0000", 2024-09-16T04:08:19.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7800000000000001e-07 2024-09-16T04:08:19.530 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.530 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096074+0000", 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3694e-05 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096103+0000", 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9322000000000001e-05 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.531 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096112+0000", 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5939999999999994e-06 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096174+0000", 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2180000000000004e-05 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.096183+0000", 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5289999999999997e-06 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.532 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102018+0000", 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058349539999999998 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102043+0000", 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5021999999999999e-05 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:19.533 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.405941293, 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083578109999999997, 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102824+0000", 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4899999999999995e-07 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102859+0000", 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4805999999999998e-05 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102891+0000", 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2394000000000002e-05 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102898+0000", 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.204e-06 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103849+0000", 2024-09-16T04:08:19.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00095089899999999995 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103858+0000", 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3480000000000004e-06 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111149+0000", 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072911809999999999 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.537 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111181+0000", 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2329999999999997e-05 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.105289+0000", 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.403475459999999, 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065227080000000003, 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105289+0000", 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105289+0000", 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105290+0000", 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.99e-07 2024-09-16T04:08:19.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106024+0000", 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00073392599999999998 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106050+0000", 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6239000000000001e-05 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106058+0000", 2024-09-16T04:08:19.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1429999999999998e-06 2024-09-16T04:08:19.541 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.541 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106110+0000", 2024-09-16T04:08:19.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1788e-05 2024-09-16T04:08:19.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106118+0000", 2024-09-16T04:08:19.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.225e-06 2024-09-16T04:08:19.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111780+0000", 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056617070000000002 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111812+0000", 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1980999999999998e-05 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.543 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 33/27 ECSubWrite(tid=35, reqid=client.4440.0:212, at_version=33'21, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.118491+0000", 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.390273611, 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049879870000000001, 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118491+0000", 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118491+0000", 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118492+0000", 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118540+0000", 2024-09-16T04:08:19.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8677000000000002e-05 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118574+0000", 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3420000000000002e-05 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118583+0000", 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2119999999999992e-06 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118637+0000", 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4243000000000002e-05 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118647+0000", 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0176e-05 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123441+0000", 2024-09-16T04:08:19.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047938859999999998 2024-09-16T04:08:19.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123479+0000", 2024-09-16T04:08:19.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.769e-05 2024-09-16T04:08:19.548 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.549 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.549 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.549 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.549 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.549 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.5s7 37/27 ECSubWrite(tid=73, reqid=osd.5.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=37'24))", 2024-09-16T04:08:19.549 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.110813+0000", 2024-09-16T04:08:19.550 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.3979515889999998, 2024-09-16T04:08:19.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006035042, 2024-09-16T04:08:19.550 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.550 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.550 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.550 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110813+0000", 2024-09-16T04:08:19.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.551 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.551 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.551 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110813+0000", 2024-09-16T04:08:19.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110813+0000", 2024-09-16T04:08:19.552 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.44e-07 2024-09-16T04:08:19.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.553 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110823+0000", 2024-09-16T04:08:19.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0529999999999996e-06 2024-09-16T04:08:19.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.553 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.553 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110850+0000", 2024-09-16T04:08:19.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7903e-05 2024-09-16T04:08:19.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.554 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.554 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.554 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110858+0000", 2024-09-16T04:08:19.554 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0250000000000002e-06 2024-09-16T04:08:19.554 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.554 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.554 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.554 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110903+0000", 2024-09-16T04:08:19.555 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4394999999999998e-05 2024-09-16T04:08:19.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.555 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.555 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.555 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110909+0000", 2024-09-16T04:08:19.555 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5150000000000003e-06 2024-09-16T04:08:19.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.555 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116812+0000", 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005903073 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116848+0000", 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5534000000000002e-05 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.556 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.4s4 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111420+0000", 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.3973449550000003, 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053788569999999999, 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.557 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.558 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111420+0000", 2024-09-16T04:08:19.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111420+0000", 2024-09-16T04:08:19.559 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.559 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.559 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111420+0000", 2024-09-16T04:08:19.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1500000000000001e-07 2024-09-16T04:08:19.561 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.562 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.562 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.562 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111442+0000", 2024-09-16T04:08:19.562 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2195999999999999e-05 2024-09-16T04:08:19.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.562 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.562 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.562 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111473+0000", 2024-09-16T04:08:19.562 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0548e-05 2024-09-16T04:08:19.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.563 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111480+0000", 2024-09-16T04:08:19.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6559999999999992e-06 2024-09-16T04:08:19.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.564 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111529+0000", 2024-09-16T04:08:19.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8828000000000001e-05 2024-09-16T04:08:19.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.564 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.564 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111538+0000", 2024-09-16T04:08:19.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7199999999999995e-06 2024-09-16T04:08:19.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.565 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.565 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116767+0000", 2024-09-16T04:08:19.565 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005229666 2024-09-16T04:08:19.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.565 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.565 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116798+0000", 2024-09-16T04:08:19.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1028000000000002e-05 2024-09-16T04:08:19.566 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.566 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.566 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.566 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 37/27 ECSubWrite(tid=128, reqid=osd.3.0:124, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:19.566 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111690+0000", 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.3970741200000001, 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051156070000000003, 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111690+0000", 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111690+0000", 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111691+0000", 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111702+0000", 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0519e-05 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111730+0000", 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8305000000000001e-05 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111739+0000", 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7530000000000007e-06 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111789+0000", 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9972000000000002e-05 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111797+0000", 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6740000000000003e-06 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116774+0000", 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049763660000000003 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116806+0000", 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2168000000000001e-05 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs3 37/27 ECSubWrite(tid=74, reqid=osd.5.0:72, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112863+0000", 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.3959014759999997, 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062791490000000004, 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.571 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112863+0000", 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112863+0000", 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112864+0000", 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4900000000000004e-07 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112875+0000", 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1112e-05 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.574 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112905+0000", 2024-09-16T04:08:19.574 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0142999999999999e-05 2024-09-16T04:08:19.574 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.574 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.574 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.574 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112913+0000", 2024-09-16T04:08:19.575 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9969999999999998e-06 2024-09-16T04:08:19.575 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.575 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.575 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.575 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112967+0000", 2024-09-16T04:08:19.575 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3928999999999998e-05 2024-09-16T04:08:19.575 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112976+0000", 2024-09-16T04:08:19.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.003e-06 2024-09-16T04:08:19.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119105+0000", 2024-09-16T04:08:19.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061291360000000003 2024-09-16T04:08:19.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119142+0000", 2024-09-16T04:08:19.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6879999999999999e-05 2024-09-16T04:08:19.577 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.577 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.578 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.578 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 37/27 ECSubWrite(tid=129, reqid=osd.3.0:129, at_version=37'43, trim_to=0'0, roll_forward_to=35'41))", 2024-09-16T04:08:19.578 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112971+0000", 2024-09-16T04:08:19.578 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.3957933300000001, 2024-09-16T04:08:19.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061749250000000004, 2024-09-16T04:08:19.579 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.579 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.579 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.579 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.579 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.579 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.580 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112971+0000", 2024-09-16T04:08:19.580 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.580 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.580 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.580 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.580 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112971+0000", 2024-09-16T04:08:19.581 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.581 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.581 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.581 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112972+0000", 2024-09-16T04:08:19.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1500000000000004e-07 2024-09-16T04:08:19.582 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.582 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.582 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.582 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112992+0000", 2024-09-16T04:08:19.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9856000000000001e-05 2024-09-16T04:08:19.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.583 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.583 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.113099+0000", 2024-09-16T04:08:19.583 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107217 2024-09-16T04:08:19.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.113107+0000", 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1759999999999993e-06 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.113322+0000", 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021465500000000001 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.113332+0000", 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7059999999999999e-06 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119094+0000", 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057622419999999999 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119146+0000", 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2358000000000003e-05 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/27 ECSubWrite(tid=155, reqid=osd.3.0:155, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439852+0000", 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.0689126789999999, 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070272690000000001, 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439852+0000", 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439852+0000", 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439852+0000", 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0200000000000001e-07 2024-09-16T04:08:19.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439882+0000", 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0241999999999999e-05 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439947+0000", 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4238000000000004e-05 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439954+0000", 2024-09-16T04:08:19.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7360000000000001e-06 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440364+0000", 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040977599999999999 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440374+0000", 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5629999999999998e-06 2024-09-16T04:08:19.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446824+0000", 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064499730000000003 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446879+0000", 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5538999999999998e-05 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.593 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 42/27 ECSubWrite(tid=50, reqid=osd.11.0:50, at_version=42'32, trim_to=0'0, roll_forward_to=41'30))", 2024-09-16T04:08:19.593 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.442356+0000", 2024-09-16T04:08:19.593 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.0664084319999998, 2024-09-16T04:08:19.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049517909999999997, 2024-09-16T04:08:19.593 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.593 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.594 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.594 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.594 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.594 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.594 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442356+0000", 2024-09-16T04:08:19.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.595 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.595 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442356+0000", 2024-09-16T04:08:19.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442356+0000", 2024-09-16T04:08:19.596 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.85e-07 2024-09-16T04:08:19.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442391+0000", 2024-09-16T04:08:19.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5209999999999997e-05 2024-09-16T04:08:19.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442437+0000", 2024-09-16T04:08:19.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5562999999999997e-05 2024-09-16T04:08:19.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442450+0000", 2024-09-16T04:08:19.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3285000000000001e-05 2024-09-16T04:08:19.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442607+0000", 2024-09-16T04:08:19.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000156787 2024-09-16T04:08:19.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.442619+0000", 2024-09-16T04:08:19.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1632e-05 2024-09-16T04:08:19.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447264+0000", 2024-09-16T04:08:19.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0046455070000000001 2024-09-16T04:08:19.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447308+0000", 2024-09-16T04:08:19.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3621999999999997e-05 2024-09-16T04:08:19.603 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.603 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.604 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.604 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:19.605 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443220+0000", 2024-09-16T04:08:19.605 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.0655446839999998, 2024-09-16T04:08:19.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064092510000000004, 2024-09-16T04:08:19.605 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.605 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.605 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.606 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.606 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.606 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443220+0000", 2024-09-16T04:08:19.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.606 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443220+0000", 2024-09-16T04:08:19.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443221+0000", 2024-09-16T04:08:19.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9999999999999995e-07 2024-09-16T04:08:19.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443544+0000", 2024-09-16T04:08:19.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00032279200000000002 2024-09-16T04:08:19.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443573+0000", 2024-09-16T04:08:19.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9546000000000001e-05 2024-09-16T04:08:19.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443580+0000", 2024-09-16T04:08:19.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4849999999999997e-06 2024-09-16T04:08:19.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.611 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443642+0000", 2024-09-16T04:08:19.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2081000000000004e-05 2024-09-16T04:08:19.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443652+0000", 2024-09-16T04:08:19.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0209e-05 2024-09-16T04:08:19.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449580+0000", 2024-09-16T04:08:19.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059278250000000003 2024-09-16T04:08:19.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449629+0000", 2024-09-16T04:08:19.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9313e-05 2024-09-16T04:08:19.613 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.614 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.614 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.614 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s2 42/27 ECSubWrite(tid=40, reqid=osd.0.0:40, at_version=42'11, trim_to=0'0, roll_forward_to=41'9))", 2024-09-16T04:08:19.614 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443756+0000", 2024-09-16T04:08:19.615 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.065008921, 2024-09-16T04:08:19.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0052538899999999998, 2024-09-16T04:08:19.615 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.615 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.615 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.615 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.616 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.616 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443756+0000", 2024-09-16T04:08:19.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443756+0000", 2024-09-16T04:08:19.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443756+0000", 2024-09-16T04:08:19.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5200000000000001e-07 2024-09-16T04:08:19.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443776+0000", 2024-09-16T04:08:19.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0026000000000001e-05 2024-09-16T04:08:19.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443811+0000", 2024-09-16T04:08:19.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5197999999999997e-05 2024-09-16T04:08:19.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443821+0000", 2024-09-16T04:08:19.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9080000000000007e-06 2024-09-16T04:08:19.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443899+0000", 2024-09-16T04:08:19.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8261000000000002e-05 2024-09-16T04:08:19.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.622 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443909+0000", 2024-09-16T04:08:19.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0332999999999999e-05 2024-09-16T04:08:19.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.448863+0000", 2024-09-16T04:08:19.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049537460000000002 2024-09-16T04:08:19.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449009+0000", 2024-09-16T04:08:19.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014626600000000001 2024-09-16T04:08:19.624 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.624 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.624 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.625 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:19.625 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.450278+0000", 2024-09-16T04:08:19.625 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.0584868099999998, 2024-09-16T04:08:19.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007464432, 2024-09-16T04:08:19.625 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.625 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.625 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.626 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450278+0000", 2024-09-16T04:08:19.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450278+0000", 2024-09-16T04:08:19.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450279+0000", 2024-09-16T04:08:19.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.031e-06 2024-09-16T04:08:19.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450548+0000", 2024-09-16T04:08:19.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000269659 2024-09-16T04:08:19.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450626+0000", 2024-09-16T04:08:19.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7577000000000006e-05 2024-09-16T04:08:19.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450635+0000", 2024-09-16T04:08:19.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2920000000000001e-06 2024-09-16T04:08:19.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450868+0000", 2024-09-16T04:08:19.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00023291100000000001 2024-09-16T04:08:19.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450879+0000", 2024-09-16T04:08:19.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0988999999999999e-05 2024-09-16T04:08:19.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457700+0000", 2024-09-16T04:08:19.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068211620000000004 2024-09-16T04:08:19.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457742+0000", 2024-09-16T04:08:19.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1811000000000003e-05 2024-09-16T04:08:19.633 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.633 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.633 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.634 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s2 42/27 ECSubWrite(tid=41, reqid=osd.0.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T04:08:19.634 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.451712+0000", 2024-09-16T04:08:19.634 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.0570520330000002, 2024-09-16T04:08:19.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060204450000000001, 2024-09-16T04:08:19.634 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.634 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.635 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.635 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451712+0000", 2024-09-16T04:08:19.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451712+0000", 2024-09-16T04:08:19.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451714+0000", 2024-09-16T04:08:19.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.125e-06 2024-09-16T04:08:19.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451730+0000", 2024-09-16T04:08:19.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6290000000000002e-05 2024-09-16T04:08:19.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451775+0000", 2024-09-16T04:08:19.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5154000000000002e-05 2024-09-16T04:08:19.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451785+0000", 2024-09-16T04:08:19.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5170000000000005e-06 2024-09-16T04:08:19.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451844+0000", 2024-09-16T04:08:19.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9540999999999998e-05 2024-09-16T04:08:19.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451869+0000", 2024-09-16T04:08:19.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4765000000000001e-05 2024-09-16T04:08:19.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457696+0000", 2024-09-16T04:08:19.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058270539999999999 2024-09-16T04:08:19.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457733+0000", 2024-09-16T04:08:19.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6999000000000003e-05 2024-09-16T04:08:19.642 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.642 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.642 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.642 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/27 ECSubWrite(tid=160, reqid=client.4440.0:347, at_version=42'32, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.452648+0000", 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.056116904, 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054755610000000003, 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452648+0000", 2024-09-16T04:08:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452648+0000", 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452648+0000", 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3500000000000005e-07 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452691+0000", 2024-09-16T04:08:19.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2284999999999999e-05 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452776+0000", 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5276000000000001e-05 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452785+0000", 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3659999999999998e-06 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452880+0000", 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4786999999999999e-05 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452890+0000", 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0061000000000001e-05 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.458088+0000", 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051975049999999998 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.458123+0000", 2024-09-16T04:08:19.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5445999999999999e-05 2024-09-16T04:08:19.647 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.647 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.647 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.647 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/27 ECSubWrite(tid=161, reqid=client.4440.0:348, at_version=42'33, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:19.648 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.452695+0000", 2024-09-16T04:08:19.648 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.056069769, 2024-09-16T04:08:19.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054508309999999997, 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452695+0000", 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452695+0000", 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:19.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452695+0000", 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5999999999999999e-07 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452706+0000", 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0676000000000001e-05 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452814+0000", 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108201 2024-09-16T04:08:19.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452821+0000", 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8179999999999998e-06 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.454553+0000", 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017320320000000001 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.454559+0000", 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7350000000000001e-06 2024-09-16T04:08:19.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.458127+0000", 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035688019999999998 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.458146+0000", 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8207e-05 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:19.652 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:19.653 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:19.654 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:19.654 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:19.651+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:19.654 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:19.655 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:19.655 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:19.656 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:19.656 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:19.656 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:19.657 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:19.657 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:19.657 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:19.658 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:19.658 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:19.658 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:19.666 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:19.666 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:19.687 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:19.687 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:19.708 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:19.709 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:19.755 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:19.753+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:19.773 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:19.856 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:19.855+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:19.916 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:19.916 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:19.925 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:19.966 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:19.964+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:20.058 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:20.056+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:20.068 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:20.068 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:20.077 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:20.159 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:20.156+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:20.187 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:20.187 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:20.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:20.261 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:20.258+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:20.303 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:20.303 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:20.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:20.362 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:20.360+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:20.418 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:20.419 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:20.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:08:20.431 INFO:tasks.rados.rados.0.smithi016.stdout:147: done (2 left) 2024-09-16T04:08:20.431 INFO:tasks.rados.rados.0.smithi016.stdout:148: read oid 40 snap -1 2024-09-16T04:08:20.431 INFO:tasks.rados.rados.0.smithi016.stdout:148: expect deleted 2024-09-16T04:08:20.431 INFO:tasks.rados.rados.0.smithi016.stdout:149: read oid 7 snap -1 2024-09-16T04:08:20.431 INFO:tasks.rados.rados.0.smithi016.stdout:149: expect (ObjNum 64 snap 2 seq_num 64) 2024-09-16T04:08:20.432 INFO:tasks.rados.rados.0.smithi016.stdout:150: snap_create 2024-09-16T04:08:20.432 INFO:tasks.rados.rados.0.smithi016.stdout:146: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T04:08:20.434 INFO:tasks.rados.rados.0.smithi016.stdout:145: finishing rollback tid 0 to smithi01639990-46 2024-09-16T04:08:20.434 INFO:tasks.rados.rados.0.smithi016.stdout:145: finishing rollback tid 1 to smithi01639990-46 2024-09-16T04:08:20.435 INFO:tasks.rados.rados.0.smithi016.stdout:145: finishing rollback tid 2 to smithi01639990-46 2024-09-16T04:08:20.435 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 46 v 46 (ObjNum 82 snap 7 seq_num 82) dirty exists 2024-09-16T04:08:20.443 INFO:tasks.rados.rados.0.smithi016.stdout:149: expect (ObjNum 64 snap 2 seq_num 64) 2024-09-16T04:08:20.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:08:20.462 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:20.461+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:20.548 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:20.548 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:20.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:08:20.564 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:20.561+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:20.665 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:20.663+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:20.666 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:20.667 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:20.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:08:20.766 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:20.764+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:20.784 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:20.784 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:20.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:20.881 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:20.879+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:20.944 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:20.944 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:20.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:20.967 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:20.965+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:21.068 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:21.067+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:21.100 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:21.100 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:21.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:21.169 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:21.167+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:21.215 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:21.215 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:21.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:21.270 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:21.268+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:21.331 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:21.332 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:21.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:21.371 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:21.369+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:21.430 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-16T04:08:21.432 INFO:tasks.rados.rados.0.smithi016.stdout:145: done (4 left) 2024-09-16T04:08:21.432 INFO:tasks.rados.rados.0.smithi016.stdout:146: done (3 left) 2024-09-16T04:08:21.432 INFO:tasks.rados.rados.0.smithi016.stdout:148: done (2 left) 2024-09-16T04:08:21.432 INFO:tasks.rados.rados.0.smithi016.stdout:149: done (1 left) 2024-09-16T04:08:21.433 INFO:tasks.rados.rados.0.smithi016.stdout:150: done (0 left) 2024-09-16T04:08:21.433 INFO:tasks.rados.rados.0.smithi016.stdout:151: delete oid 48 current snap is 9 2024-09-16T04:08:21.471 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:21.469+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:21.499 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:21.499 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:21.507 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:21.572 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:21.570+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:21.615 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:21.615 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:21.624 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:21.673 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:21.671+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:21.737 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:21.737 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:21.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:21.746 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:21.787 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:21.784+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:21.874 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:21.875 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:21.873+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:21.889 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:21.889 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:21.895 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:21.896 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:21.901 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:21.916 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:21.917 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:21.937 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:21.937 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:21.958 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:21.975 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:21.973+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:21.980 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:21.980 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:22.001 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:22.001 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:22.007 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:22.007 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:22.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:199 3.3s0 3:c05d4807:::smithi01639990-31:head [copy-from ver 11] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.794956990999999, 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.48371305599999997, 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:22.008 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 199 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209959+0000", 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7460000000000001e-06 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209971+0000", 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2787e-05 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210000+0000", 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8952999999999999e-05 2024-09-16T04:08:22.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210015+0000", 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.434e-05 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210169+0000", 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015455199999999999 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210317+0000", 2024-09-16T04:08:22.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000147291 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.224854+0000", 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014536926 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226325+0000", 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001471099 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693612+0000", 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.467287324 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693669+0000", 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7037999999999998e-05 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.013 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:200 3.3s0 3:c05d4807:::smithi01639990-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.794866702, 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016139113, 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 200 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210047+0000", 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210051+0000", 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8800000000000001e-06 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210065+0000", 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3567000000000001e-05 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210074+0000", 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2399999999999996e-06 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210464+0000", 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00038978800000000002 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210496+0000", 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2539999999999997e-05 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.225987+0000", 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015491398999999999 2024-09-16T04:08:22.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226084+0000", 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6618000000000002e-05 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226185+0000", 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101205 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.018 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s6 33/27 ECSubWrite(tid=37, reqid=osd.8.0:37, at_version=33'25, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.098670+0000", 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.906242948999999, 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058761789999999996, 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098670+0000", 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098670+0000", 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098670+0000", 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2700000000000001e-07 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098688+0000", 2024-09-16T04:08:22.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7703e-05 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098726+0000", 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7818000000000003e-05 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098735+0000", 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9109999999999999e-06 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098816+0000", 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1668999999999993e-05 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098825+0000", 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2979999999999997e-06 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104517+0000", 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0056911380000000001 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104546+0000", 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9515000000000001e-05 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s6 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:22.023 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.905322763000001, 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010829642, 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099591+0000", 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4399999999999999e-07 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099619+0000", 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7555e-05 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099655+0000", 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6337000000000001e-05 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099691+0000", 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5994000000000001e-05 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099819+0000", 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000128484 2024-09-16T04:08:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099829+0000", 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1949999999999992e-06 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110366+0000", 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010537504 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110420+0000", 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3729e-05 2024-09-16T04:08:22.027 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s2 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.903506131, 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012298224, 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:08:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000002e-07 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101422+0000", 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5004000000000001e-05 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101446+0000", 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3632999999999999e-05 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101452+0000", 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0260000000000004e-06 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101511+0000", 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9515000000000002e-05 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101523+0000", 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1094000000000001e-05 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113680+0000", 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012157401 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113705+0000", 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5041999999999998e-05 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.es3 33/27 ECSubWrite(tid=31, reqid=osd.11.0:31, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-16T04:08:22.032 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.903387780999999, 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012148897000000001, 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:08:22.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101526+0000", 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0900000000000001e-07 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101543+0000", 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7215999999999999e-05 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101584+0000", 2024-09-16T04:08:22.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1352e-05 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101590+0000", 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5779999999999998e-06 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101650+0000", 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0093000000000002e-05 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101662+0000", 2024-09-16T04:08:22.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2257e-05 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113631+0000", 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.01196856 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113674+0000", 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3132e-05 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.903364764999999, 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012128856, 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:08:22.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2399999999999999e-07 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101558+0000", 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1980000000000007e-06 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101618+0000", 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0911e-05 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101624+0000", 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6899999999999997e-06 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101690+0000", 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5988000000000006e-05 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101701+0000", 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0878e-05 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113645+0000", 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011944110000000001 2024-09-16T04:08:22.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113677+0000", 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1856999999999997e-05 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.901885711, 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014214867000000001, 2024-09-16T04:08:22.041 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103028+0000", 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8999999999999995e-07 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103051+0000", 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2909999999999999e-05 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103122+0000", 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1339000000000003e-05 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103130+0000", 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8790000000000003e-06 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104942+0000", 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001811893 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104952+0000", 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5449999999999992e-06 2024-09-16T04:08:22.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117201+0000", 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012249395 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117242+0000", 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1016000000000002e-05 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.046 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:22.046 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:08:22.046 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.900358151000001, 2024-09-16T04:08:22.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012207213999999999, 2024-09-16T04:08:22.046 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.046 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.046 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.046 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104556+0000", 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:08:22.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104963+0000", 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000407604 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105023+0000", 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9684e-05 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105031+0000", 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3599999999999996e-06 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105239+0000", 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00020760900000000001 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105251+0000", 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1788e-05 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116731+0000", 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011480292 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116762+0000", 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1087000000000002e-05 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.050 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.107022+0000", 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.897891209000001, 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097676329999999995, 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107022+0000", 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107022+0000", 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107022+0000", 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2900000000000003e-07 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107282+0000", 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025955999999999999 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107317+0000", 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5176999999999999e-05 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107326+0000", 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1220000000000001e-06 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111509+0000", 2024-09-16T04:08:22.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041823099999999998 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111518+0000", 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4870000000000008e-06 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116766+0000", 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052478159999999998 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116789+0000", 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3431999999999999e-05 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=80, reqid=client.4440.0:209, at_version=33'23, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.108124+0000", 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.896789172, 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086904689999999993, 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.055 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108124+0000", 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108124+0000", 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108124+0000", 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108135+0000", 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1338000000000001e-05 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108191+0000", 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6217000000000001e-05 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108197+0000", 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8819999999999998e-06 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111944+0000", 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037468250000000001 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111953+0000", 2024-09-16T04:08:22.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7269999999999996e-06 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116794+0000", 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048407049999999998 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116814+0000", 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0602e-05 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.059 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=81, reqid=client.4440.0:202, at_version=33'25, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.113700+0000", 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.891212911, 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079098669999999992, 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113700+0000", 2024-09-16T04:08:22.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113700+0000", 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113701+0000", 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9299999999999997e-07 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113865+0000", 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016396499999999999 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113890+0000", 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5024e-05 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113897+0000", 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6090000000000003e-06 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113949+0000", 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2037999999999999e-05 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113957+0000", 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1940000000000001e-06 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121563+0000", 2024-09-16T04:08:22.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0076067779999999998 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121610+0000", 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6465999999999999e-05 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.es3 33/27 ECSubWrite(tid=33, reqid=osd.11.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=33'6))", 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.114128+0000", 2024-09-16T04:08:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.890784921, 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074696759999999997, 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114128+0000", 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114128+0000", 2024-09-16T04:08:22.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114129+0000", 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6000000000000003e-07 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114139+0000", 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0764999999999999e-05 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114164+0000", 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5001999999999999e-05 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114170+0000", 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9000000000000003e-06 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114215+0000", 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4308000000000003e-05 2024-09-16T04:08:22.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114221+0000", 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0430000000000004e-06 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121555+0000", 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073344179999999997 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121598+0000", 2024-09-16T04:08:22.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2580000000000002e-05 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s2 33/27 ECSubWrite(tid=17, reqid=osd.0.0:16, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.114574+0000", 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.890338677000001, 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070510010000000003, 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114574+0000", 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114574+0000", 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114575+0000", 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3200000000000003e-07 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114585+0000", 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9140000000000003e-06 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114619+0000", 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4289999999999999e-05 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114625+0000", 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8939999999999998e-06 2024-09-16T04:08:22.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114671+0000", 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5636999999999997e-05 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114681+0000", 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0298e-05 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121602+0000", 2024-09-16T04:08:22.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069206199999999997 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121625+0000", 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3416000000000001e-05 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=34, reqid=client.4440.0:211, at_version=33'20, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.115642+0000", 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.889271032, 2024-09-16T04:08:22.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0063991539999999998, 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115642+0000", 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115642+0000", 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115643+0000", 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4200000000000005e-07 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115822+0000", 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000179273 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115863+0000", 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1133000000000001e-05 2024-09-16T04:08:22.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115870+0000", 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115941+0000", 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0547000000000001e-05 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115947+0000", 2024-09-16T04:08:22.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3810000000000004e-06 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.122004+0000", 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060568369999999998 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.122041+0000", 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7049999999999999e-05 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s3 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.141650+0000", 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.86326313, 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062685240000000001, 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141650+0000", 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141650+0000", 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141651+0000", 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5099999999999999e-07 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141667+0000", 2024-09-16T04:08:22.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6458000000000001e-05 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141697+0000", 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699000000000001e-05 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141705+0000", 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7139999999999999e-06 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142774+0000", 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00106846 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142783+0000", 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6050000000000004e-06 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.147871+0000", 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0050880200000000004 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.147918+0000", 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6817000000000003e-05 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds2 37/31 ECSubWrite(tid=163, reqid=osd.7.0:163, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.108559+0000", 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.8963541779999993, 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0064639540000000001, 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.082 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108559+0000", 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108559+0000", 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108559+0000", 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8099999999999999e-07 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108585+0000", 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6074000000000001e-05 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108646+0000", 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1333000000000004e-05 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108655+0000", 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1689999999999998e-06 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108710+0000", 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4478999999999998e-05 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108718+0000", 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6359999999999994e-06 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114968+0000", 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062499280000000001 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115023+0000", 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5154000000000001e-05 2024-09-16T04:08:22.086 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds2 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.109506+0000", 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.8954068520000007, 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.007408492, 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109506+0000", 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109506+0000", 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109507+0000", 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0200000000000001e-07 2024-09-16T04:08:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109527+0000", 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0098999999999998e-05 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109573+0000", 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5812000000000001e-05 2024-09-16T04:08:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109580+0000", 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109636+0000", 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5810000000000003e-05 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109644+0000", 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7540000000000006e-06 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116885+0000", 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0072415550000000002 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116915+0000", 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9181999999999999e-05 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 37/27 ECSubWrite(tid=129, reqid=osd.3.0:129, at_version=37'43, trim_to=0'0, roll_forward_to=35'41))", 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.112730+0000", 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.8921830630000009, 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0055167389999999997, 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112730+0000", 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112730+0000", 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112730+0000", 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.44e-07 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112748+0000", 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7589999999999999e-05 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112782+0000", 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3547000000000003e-05 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112792+0000", 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0689999999999999e-05 2024-09-16T04:08:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112854+0000", 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1410999999999999e-05 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112864+0000", 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0482000000000001e-05 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118215+0000", 2024-09-16T04:08:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0053510529999999997 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118247+0000", 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1322000000000003e-05 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s3 42/27 ECSubWrite(tid=160, reqid=client.4440.0:347, at_version=42'32, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.450770+0000", 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5541433490000003, 2024-09-16T04:08:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0055173879999999998, 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450770+0000", 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450770+0000", 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450770+0000", 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8299999999999995e-07 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450799+0000", 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8738000000000002e-05 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450836+0000", 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6331000000000002e-05 2024-09-16T04:08:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450844+0000", 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6289999999999999e-06 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450901+0000", 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7306000000000003e-05 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450911+0000", 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1120000000000002e-06 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456257+0000", 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0053461150000000002 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456287+0000", 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0273999999999998e-05 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.101 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:22.102 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:22.104 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:22.104 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:22.105 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:22.105 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:22.106 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:22.099+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:22.106 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:22.106 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:22.106 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:22.107 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:22.109 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:22.110 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:22.203 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:22.201+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:22.248 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:22.248 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:22.248 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:22.248 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:22.256 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:22.304 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:22.301+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:22.359 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:22.359 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:22.359 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:22.359 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:22.359 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:22.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:22.408 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:22.406+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:22.465 INFO:tasks.rados.rados.0.smithi016.stdout:151: done (0 left) 2024-09-16T04:08:22.465 INFO:tasks.rados.rados.0.smithi016.stdout:152: snap_remove snap 7 2024-09-16T04:08:22.506 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:22.505+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:22.526 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:22.526 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:22.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:22.526 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:22.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.526 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:185 3.5s0 3:a7f99973:::smithi01639990-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:22.526 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.185094+0000", 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.338531014999999, 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077979190000000004, 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 185 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185094+0000", 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185094+0000", 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185095+0000", 2024-09-16T04:08:22.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7299999999999996e-07 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185098+0000", 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4679999999999999e-06 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185104+0000", 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5189999999999998e-06 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185109+0000", 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.057e-06 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.188386+0000", 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032768250000000001 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.188443+0000", 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7426000000000002e-05 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191927+0000", 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034834190000000002 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.192006+0000", 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9634000000000004e-05 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.192892+0000", 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088599800000000004 2024-09-16T04:08:22.531 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 33/27 ECSubWrite(tid=37, reqid=osd.8.0:37, at_version=33'25, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.424918781000001, 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088041119999999994, 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7000000000000001e-07 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098728+0000", 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1432000000000001e-05 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098770+0000", 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1486999999999998e-05 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098780+0000", 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7280000000000007e-06 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098852+0000", 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2461999999999998e-05 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098862+0000", 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9369999999999999e-06 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107472+0000", 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086102990000000001 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107511+0000", 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8497000000000001e-05 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:22.536 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.423748169, 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011088002, 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099878+0000", 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8400000000000003e-07 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099902+0000", 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3629999999999999e-05 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.538 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099934+0000", 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2388999999999998e-05 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099943+0000", 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4800000000000001e-06 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.100212+0000", 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026930999999999998 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.100220+0000", 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8339999999999999e-06 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110928+0000", 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010708333 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110965+0000", 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7141999999999998e-05 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.540 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s7 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.421535849, 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088092050000000005, 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.18e-07 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102106+0000", 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6116999999999998e-05 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102134+0000", 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7963e-05 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102140+0000", 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:22.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102224+0000", 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3634000000000006e-05 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102233+0000", 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1609999999999992e-06 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110868+0000", 2024-09-16T04:08:22.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086352040000000005 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110899+0000", 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0852999999999998e-05 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.420610013999999, 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013925085, 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:22.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103016+0000", 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5499999999999999e-07 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103038+0000", 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2416e-05 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103157+0000", 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000118439 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103166+0000", 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9819999999999997e-06 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103225+0000", 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8996000000000002e-05 2024-09-16T04:08:22.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103235+0000", 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9909999999999997e-06 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116899+0000", 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013664549999999999 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116941+0000", 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1455999999999997e-05 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.420584713, 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013929731000000001, 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.550 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:22.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5899999999999998e-07 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103057+0000", 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5778e-05 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103193+0000", 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0001359 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103199+0000", 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1279999999999996e-06 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107717+0000", 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045183769999999996 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107726+0000", 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5420000000000005e-06 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116950+0000", 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092240229999999996 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116970+0000", 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0724e-05 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s5 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:22.554 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.419835911, 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070187460000000002, 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103801+0000", 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0927999999999999e-05 2024-09-16T04:08:22.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103842+0000", 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0649999999999999e-05 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103851+0000", 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8920000000000006e-06 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103944+0000", 2024-09-16T04:08:22.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2705000000000001e-05 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103955+0000", 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1282e-05 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110771+0000", 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0068159270000000003 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110808+0000", 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7512000000000003e-05 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(osd.3.19:2 3.cs0 3:3d7a8735:::smithi01639990-20:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105118+0000", 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.418507759000001, 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0066834290000000003, 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.559 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105118+0000", 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105118+0000", 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105120+0000", 2024-09-16T04:08:22.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.855e-06 2024-09-16T04:08:22.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105129+0000", 2024-09-16T04:08:22.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2739999999999996e-06 2024-09-16T04:08:22.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105164+0000", 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5621000000000003e-05 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105182+0000", 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7790000000000001e-05 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107894+0000", 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027115860000000002 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107957+0000", 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3202999999999998e-05 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111801+0000", 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038441 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.563 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 33/27 ECSubWrite(tid=85, reqid=client.4440.0:216, at_version=33'15, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.140827+0000", 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.382798321999999, 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069260850000000002, 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140827+0000", 2024-09-16T04:08:22.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140827+0000", 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140828+0000", 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1900000000000001e-07 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141044+0000", 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00021635099999999999 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141081+0000", 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6628000000000002e-05 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141087+0000", 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.297e-06 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141146+0000", 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8838999999999997e-05 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141157+0000", 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1105e-05 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.147715+0000", 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065581980000000003 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.147753+0000", 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7747999999999998e-05 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s5 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4167210230000009, 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0080941920000000001, 2024-09-16T04:08:22.568 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106905+0000", 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4999999999999999e-07 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106926+0000", 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1202000000000001e-05 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107015+0000", 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9407999999999995e-05 2024-09-16T04:08:22.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107034+0000", 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8896999999999999e-05 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107499+0000", 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046474800000000002 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107509+0000", 2024-09-16T04:08:22.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0390999999999999e-05 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114906+0000", 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073969409999999998 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114999+0000", 2024-09-16T04:08:22.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2454999999999995e-05 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds7 37/31 ECSubWrite(tid=163, reqid=osd.7.0:163, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4150678200000009, 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084527260000000007, 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0100000000000004e-07 2024-09-16T04:08:22.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108578+0000", 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9366999999999999e-05 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108616+0000", 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8056999999999998e-05 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108624+0000", 2024-09-16T04:08:22.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.824e-06 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108684+0000", 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0019999999999998e-05 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108693+0000", 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2599999999999994e-06 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116974+0000", 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082815990000000006 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.117010+0000", 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5898000000000001e-05 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.577 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds7 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4140939160000006, 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091377019999999993, 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6499999999999998e-07 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109554+0000", 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2030000000000001e-05 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109587+0000", 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3019999999999999e-05 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109603+0000", 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5678000000000001e-05 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110514+0000", 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00091074700000000005 2024-09-16T04:08:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110523+0000", 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4639999999999995e-06 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118636+0000", 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008112157 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118669+0000", 2024-09-16T04:08:22.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3741000000000001e-05 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.4122108890000007, 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0072197169999999996, 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7400000000000003e-07 2024-09-16T04:08:22.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111431+0000", 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5305e-05 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111457+0000", 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6067e-05 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.584 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111462+0000", 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3480000000000003e-06 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111516+0000", 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3897000000000003e-05 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111526+0000", 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7440000000000002e-06 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118592+0000", 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070660710000000002 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118634+0000", 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2611000000000002e-05 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.586 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.0791534970000001, 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011544518, 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444473+0000", 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4499999999999996e-07 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447203+0000", 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002729834 2024-09-16T04:08:22.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447277+0000", 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4078999999999993e-05 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447283+0000", 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4169999999999997e-06 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447458+0000", 2024-09-16T04:08:22.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00017450300000000001 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447469+0000", 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1062e-05 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.455978+0000", 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085090730000000007 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456017+0000", 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8804999999999999e-05 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:350 3.cs0 3:338c6e39:::smithi01639990-33:6 [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.446406+0000", 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.0772194700000002, 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00699593, 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.591 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 350 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446406+0000", 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446406+0000", 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446408+0000", 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6419999999999999e-06 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446414+0000", 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2689999999999996e-06 2024-09-16T04:08:22.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446425+0000", 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.102e-05 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446432+0000", 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5020000000000003e-06 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446537+0000", 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000105035 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446647+0000", 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109121 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453402+0000", 2024-09-16T04:08:22.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067553409999999998 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:351 3.cs0 3:338c6e39:::smithi01639990-33:6 [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.446452+0000", 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.0771735749999998, 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074593050000000003, 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.596 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 351 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446452+0000", 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446452+0000", 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446452+0000", 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2300000000000001e-07 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446456+0000", 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.242e-06 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446462+0000", 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6329999999999999e-06 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446475+0000", 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2439999999999999e-05 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446992+0000", 2024-09-16T04:08:22.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00051762900000000003 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447118+0000", 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012605900000000001 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453911+0000", 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067927789999999997 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.600 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:352 3.cs0 3:338c6e39:::smithi01639990-33:6 [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.0771416389999997, 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078540529999999997, 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:22.601 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 352 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9200000000000001e-07 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446487+0000", 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0489999999999999e-06 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446494+0000", 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5180000000000001e-06 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446499+0000", 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9010000000000002e-06 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447665+0000", 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0011659120000000001 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447747+0000", 2024-09-16T04:08:22.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2785000000000001e-05 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.454338+0000", 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065903960000000001 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 42/27 ECSubWrite(tid=51, reqid=osd.11.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=42'32))", 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.448304+0000", 2024-09-16T04:08:22.605 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.0753216800000001, 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077603150000000003, 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448304+0000", 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448304+0000", 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448305+0000", 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4300000000000002e-07 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448319+0000", 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4438e-05 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448347+0000", 2024-09-16T04:08:22.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8070999999999999e-05 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448357+0000", 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7960000000000007e-06 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448412+0000", 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5300000000000002e-05 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.608 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448506+0000", 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3948000000000002e-05 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456027+0000", 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075212589999999998 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456064+0000", 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.676e-05 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.609 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/27 ECSubWriteReply(tid=88, last_complete=42'39, committed=1, applied=1))", 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.0751794800000001, 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49630058100000002, 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:08:22.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448447+0000", 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448451+0000", 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3030000000000001e-06 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944646+0000", 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49619510999999999 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944655+0000", 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8480000000000007e-06 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944717+0000", 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2065999999999998e-05 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944747+0000", 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9505999999999998e-05 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.450032+0000", 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.0735930050000002, 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088454970000000008, 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450032+0000", 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450032+0000", 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450033+0000", 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2399999999999998e-07 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451065+0000", 2024-09-16T04:08:22.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010323089999999999 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451134+0000", 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8096999999999995e-05 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451140+0000", 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5939999999999999e-06 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451197+0000", 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6665e-05 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451206+0000", 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3530000000000001e-06 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.458843+0000", 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0076371549999999996 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.458878+0000", 2024-09-16T04:08:22.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4700000000000003e-05 2024-09-16T04:08:22.618 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.618 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.618 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.618 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.618 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.618 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:22.620 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:22.620 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:22.618+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:22.721 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:22.719+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:22.725 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:22.728 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:22.728 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:22.728 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:22.734 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:22.822 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:22.821+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:22.837 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:22.838 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:22.838 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:22.838 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:22.838 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:22.846 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:22.923 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:22.922+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:22.951 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:22.951 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:22.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:22.951 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:22.951 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.951 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:193 3.as0 3:5d02464a:::smithi01639990-40:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:22.951 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:22.951 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.751981153999999, 2024-09-16T04:08:22.951 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041575429999999997, 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 193 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:22.952 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196589+0000", 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8160000000000001e-06 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.953 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196600+0000", 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1545e-05 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196613+0000", 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.34e-05 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196621+0000", 2024-09-16T04:08:22.954 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4909999999999999e-06 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196682+0000", 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1074000000000005e-05 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196776+0000", 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4102000000000001e-05 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.955 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.197563+0000", 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000786446 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.198374+0000", 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081128199999999998 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200668+0000", 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0022936979999999998 2024-09-16T04:08:22.956 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200744+0000", 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6688999999999997e-05 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.844864475, 2024-09-16T04:08:22.957 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0068480019999999997, 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.958 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103704+0000", 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7100000000000001e-07 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103727+0000", 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3694999999999999e-05 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.959 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103810+0000", 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3091999999999997e-05 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103816+0000", 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4519999999999996e-06 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.960 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103971+0000", 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000154993 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103982+0000", 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0933e-05 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110491+0000", 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065096800000000003 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.961 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110551+0000", 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9985999999999998e-05 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.844815136999999, 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067325809999999996, 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.962 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.963 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103754+0000", 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5899999999999995e-07 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103768+0000", 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4562999999999999e-05 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103805+0000", 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6730000000000002e-05 2024-09-16T04:08:22.964 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103813+0000", 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103868+0000", 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4475000000000003e-05 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103877+0000", 2024-09-16T04:08:22.965 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7099999999999996e-06 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110444+0000", 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065675170000000001 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110485+0000", 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1177000000000002e-05 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.966 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/27 ECSubWrite(tid=84, reqid=client.4440.0:215, at_version=33'14, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.809855517999999, 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042300860000000001, 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:22.967 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5900000000000001e-07 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.968 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139362+0000", 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064961800000000003 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139387+0000", 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5369999999999999e-05 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139394+0000", 2024-09-16T04:08:22.969 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2190000000000001e-06 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139458+0000", 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4603000000000005e-05 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139470+0000", 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1810999999999999e-05 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.970 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142910+0000", 2024-09-16T04:08:22.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003439896 2024-09-16T04:08:22.971 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.971 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.971 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.971 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142942+0000", 2024-09-16T04:08:22.971 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2410000000000003e-05 2024-09-16T04:08:22.971 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.971 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.971 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.972 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.972 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.972 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/27 ECSubWrite(tid=85, reqid=client.4440.0:216, at_version=33'15, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:22.972 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:22.972 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.807653245000001, 2024-09-16T04:08:22.972 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043866510000000001, 2024-09-16T04:08:22.972 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.972 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.972 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:22.973 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141225+0000", 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00030972200000000001 2024-09-16T04:08:22.974 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141258+0000", 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3228000000000001e-05 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141267+0000", 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5590000000000006e-06 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141327+0000", 2024-09-16T04:08:22.975 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0100999999999999e-05 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141338+0000", 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0745e-05 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145260+0000", 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039227610000000003 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.976 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145301+0000", 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0884999999999999e-05 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 37/27 ECSubWrite(tid=71, reqid=osd.5.0:71, at_version=37'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8458773260000001, 2024-09-16T04:08:22.977 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040595509999999998, 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.978 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4000000000000001e-07 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102711+0000", 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0409999999999999e-05 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.979 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103362+0000", 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065125100000000004 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103372+0000", 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7340000000000003e-06 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103464+0000", 2024-09-16T04:08:22.980 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1663999999999995e-05 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103474+0000", 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8630000000000003e-06 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106711+0000", 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032372669999999998 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.981 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106750+0000", 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9221999999999998e-05 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=70, reqid=osd.5.0:70, at_version=37'20, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8458552049999994, 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0053540310000000004, 2024-09-16T04:08:22.982 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3799999999999999e-07 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102721+0000", 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6500000000000002e-06 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.984 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103774+0000", 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010529070000000001 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103806+0000", 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1303999999999997e-05 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103868+0000", 2024-09-16T04:08:22.985 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1937e-05 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103877+0000", 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6199999999999994e-06 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108037+0000", 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041593389999999997 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.986 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108067+0000", 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0136000000000001e-05 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/27 ECSubWrite(tid=124, reqid=osd.3.0:124, at_version=37'22, trim_to=0'0, roll_forward_to=35'20))", 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8452308370000008, 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070925160000000001, 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.987 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.988 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6999999999999999e-07 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103359+0000", 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1835999999999999e-05 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.989 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103683+0000", 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00032427399999999999 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103693+0000", 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7380000000000006e-06 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103756+0000", 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3115999999999997e-05 2024-09-16T04:08:22.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103766+0000", 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0043e-05 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110407+0000", 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0066407230000000003 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110430+0000", 2024-09-16T04:08:22.991 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2616000000000002e-05 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8444005640000007, 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008202661, 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.992 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.993 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104168+0000", 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0200000000000001e-07 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104188+0000", 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9933999999999999e-05 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104232+0000", 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4196000000000002e-05 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.994 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104241+0000", 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7979999999999995e-06 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104928+0000", 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687124 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104937+0000", 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5269999999999998e-06 2024-09-16T04:08:22.995 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112328+0000", 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073914510000000003 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112370+0000", 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1928999999999998e-05 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:22.996 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.bs7 37/27 ECSubWrite(tid=127, reqid=osd.3.0:127, at_version=37'22, trim_to=0'0, roll_forward_to=35'17))", 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.105041+0000", 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8435266759999998, 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046635560000000001, 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105041+0000", 2024-09-16T04:08:22.997 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105041+0000", 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105042+0000", 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9299999999999997e-07 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105073+0000", 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1041999999999999e-05 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105110+0000", 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7113000000000002e-05 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105118+0000", 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4330000000000004e-06 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105253+0000", 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000134191 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105264+0000", 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1827e-05 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109656+0000", 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043918459999999996 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109705+0000", 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8411000000000001e-05 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8416809559999994, 2024-09-16T04:08:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082659529999999995, 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1600000000000002e-07 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106910+0000", 2024-09-16T04:08:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2954000000000001e-05 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106990+0000", 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9788000000000002e-05 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106997+0000", 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4379999999999998e-06 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107267+0000", 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026999599999999999 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107276+0000", 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9530000000000005e-06 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115126+0000", 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078491940000000003 2024-09-16T04:08:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115153+0000", 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7214000000000001e-05 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 37/27 ECSubWrite(tid=73, reqid=osd.5.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=37'24))", 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.110673+0000", 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8378948039999994, 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044773169999999998, 2024-09-16T04:08:23.006 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110673+0000", 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.007 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110673+0000", 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110674+0000", 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6199999999999997e-07 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110685+0000", 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1221e-05 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.008 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110720+0000", 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5049000000000003e-05 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110729+0000", 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3370000000000006e-06 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110800+0000", 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0425000000000004e-05 2024-09-16T04:08:23.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110810+0000", 2024-09-16T04:08:23.010 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-05 2024-09-16T04:08:23.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.010 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.010 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.010 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115120+0000", 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043095920000000001 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115150+0000", 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0840999999999998e-05 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.011 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.4s6 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8371949870000002, 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0054036320000000002, 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.012 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899999999999999e-07 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111387+0000", 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3501e-05 2024-09-16T04:08:23.013 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111416+0000", 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9377999999999999e-05 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111421+0000", 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2889999999999997e-06 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.014 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111470+0000", 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9162000000000001e-05 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111479+0000", 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9900000000000003e-06 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.015 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116730+0000", 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052506219999999999 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116777+0000", 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6551000000000002e-05 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.016 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=74, reqid=osd.5.0:72, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8357681780000004, 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040065320000000001, 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.017 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5400000000000001e-07 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112809+0000", 2024-09-16T04:08:23.018 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9050000000000003e-06 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112841+0000", 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1983999999999997e-05 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112849+0000", 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3750000000000003e-06 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.019 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112915+0000", 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5599000000000007e-05 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112922+0000", 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116782+0000", 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038605890000000002 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.020 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116806+0000", 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3760999999999998e-05 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.498627731, 2024-09-16T04:08:23.021 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091491550000000008, 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.022 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5800000000000001e-07 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450512+0000", 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057160000000000002 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450541+0000", 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8835000000000001e-05 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450549+0000", 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3080000000000008e-06 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450607+0000", 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7763999999999998e-05 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450616+0000", 2024-09-16T04:08:23.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459045+0000", 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084290709999999998 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459089+0000", 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4233000000000002e-05 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.025 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/27 ECSubWrite(tid=41, reqid=osd.0.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.451624+0000", 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.4969435329999996, 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044678180000000001, 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451624+0000", 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451624+0000", 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451625+0000", 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7100000000000001e-07 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451635+0000", 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.825e-06 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451672+0000", 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7134000000000001e-05 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451680+0000", 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.028 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451737+0000", 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6675000000000002e-05 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451744+0000", 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.041e-06 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456050+0000", 2024-09-16T04:08:23.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043059860000000004 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456092+0000", 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2014999999999997e-05 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=160, reqid=client.4440.0:347, at_version=42'32, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:23.030 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.4960056489999998, 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065520760000000004, 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452563+0000", 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6300000000000005e-07 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452621+0000", 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8488e-05 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452654+0000", 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2750999999999999e-05 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452659+0000", 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.293e-06 2024-09-16T04:08:23.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452730+0000", 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0173999999999996e-05 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452738+0000", 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6260000000000001e-06 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.034 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459095+0000", 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0063566530000000003 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459114+0000", 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9428000000000001e-05 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=161, reqid=client.4440.0:348, at_version=42'33, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:23.035 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.4959430429999996, 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065540390000000002, 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6300000000000001e-07 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452632+0000", 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6699999999999997e-06 2024-09-16T04:08:23.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452678+0000", 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5760000000000002e-05 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452682+0000", 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.036e-06 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453280+0000", 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059861699999999996 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453289+0000", 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9590000000000001e-06 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459117+0000", 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058281770000000004 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459179+0000", 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1556999999999994e-05 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=167, reqid=client.4440.0:355, at_version=42'37, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.588912+0000", 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.3596554879999996, 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004899128, 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.040 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.588912+0000", 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.588912+0000", 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.588913+0000", 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4199999999999995e-07 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589099+0000", 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018556200000000001 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589134+0000", 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5553999999999999e-05 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589142+0000", 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0979999999999994e-06 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589199+0000", 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6586999999999999e-05 2024-09-16T04:08:23.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589210+0000", 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.078e-05 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.593773+0000", 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045635179999999999 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.593811+0000", 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8287000000000001e-05 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:362 3.as0 3:5e52341f:::smithi01639990-38:head [] snapc 8=[6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:18.439764+0000", 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5088037910000001, 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004077855, 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.045 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 362 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439764+0000", 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439764+0000", 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439765+0000", 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2839999999999999e-06 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439771+0000", 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0580000000000003e-06 2024-09-16T04:08:23.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439784+0000", 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2301e-05 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439790+0000", 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1020000000000002e-06 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439842+0000", 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2308999999999997e-05 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439848+0000", 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0469999999999999e-06 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.443795+0000", 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039468209999999997 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.443842+0000", 2024-09-16T04:08:23.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6932999999999999e-05 2024-09-16T04:08:23.050 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.050 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.050 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.050 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.050 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.050 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:23.052 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:23.052 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:23.050+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:23.153 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:23.151+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:23.155 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:23.155 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:23.155 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:23.156 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:23.165 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:23.254 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:23.252+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:23.272 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:23.272 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:23.272 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:23.272 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:23.272 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:23.280 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:23.354 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:23.352+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:23.386 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:23.386 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:23.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.442941134999998, 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.387 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:23.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:23.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:23.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:23.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:23.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:23.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.394 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:23.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:23.394 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.394 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.394 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.394 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:23.394 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 32.460536892, 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.396 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:23.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.396 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.396 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:23.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:23.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:23.401 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 32.450174386, 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:23.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.953586334000001, 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.406 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.407 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.408 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:23.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:23.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.930073432, 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092583400000000003, 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.412 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.413 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.413 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.413 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.413 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-09-16T04:08:23.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.413 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:23.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.37e-07 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453604+0000", 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2658e-05 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453611+0000", 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0650000000000001e-06 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453616+0000", 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9099999999999996e-06 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459323+0000", 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057072879999999996 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459369+0000", 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6436000000000001e-05 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459442+0000", 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2459000000000005e-05 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460087+0000", 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064517999999999997 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462796+0000", 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027087639999999998 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462827+0000", 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0827000000000001e-05 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462849+0000", 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2215999999999999e-05 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.418 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:160 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.499154179000001, 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088385849999999995, 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 160 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884510+0000", 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3900000000000005e-07 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884514+0000", 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2169999999999999e-06 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884520+0000", 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8639999999999998e-06 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884527+0000", 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.866e-06 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888915+0000", 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043874270000000002 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.888953+0000", 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7839000000000001e-05 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889014+0000", 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1289000000000002e-05 2024-09-16T04:08:23.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892771+0000", 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037575199999999999 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893289+0000", 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00051718599999999999 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893314+0000", 2024-09-16T04:08:23.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5202999999999999e-05 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893348+0000", 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4635000000000003e-05 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.499116999000002, 2024-09-16T04:08:23.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088608530000000001, 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 161 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884548+0000", 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5599999999999995e-07 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884603+0000", 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5615999999999997e-05 2024-09-16T04:08:23.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884612+0000", 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6589999999999996e-06 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884619+0000", 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4630000000000004e-06 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889203+0000", 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045841989999999997 2024-09-16T04:08:23.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889242+0000", 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8713e-05 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889304+0000", 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1858000000000003e-05 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892819+0000", 2024-09-16T04:08:23.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003515107 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893354+0000", 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053514000000000003 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893377+0000", 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2265000000000001e-05 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.429 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893408+0000", 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1276999999999999e-05 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.499037749999999, 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094233230000000008, 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.430 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 162 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:23.431 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884627+0000", 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884629+0000", 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.187e-06 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.432 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884632+0000", 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.501e-06 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884635+0000", 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7389999999999999e-06 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892223+0000", 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075881029999999997 2024-09-16T04:08:23.433 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892291+0000", 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7508000000000005e-05 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892412+0000", 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012124 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893979+0000", 2024-09-16T04:08:23.434 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015676069999999999 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893987+0000", 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.306e-06 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894013+0000", 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6305999999999999e-05 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.435 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894050+0000", 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6492999999999998e-05 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.499025378999999, 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095013200000000006, 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.436 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 163 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:23.437 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2100000000000001e-07 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884642+0000", 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.478e-06 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.438 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884646+0000", 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.292e-06 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884648+0000", 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6230000000000001e-06 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892847+0000", 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0081990580000000004 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.439 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892891+0000", 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3628999999999999e-05 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892959+0000", 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8151000000000005e-05 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894055+0000", 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001095927 2024-09-16T04:08:23.440 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894088+0000", 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3016999999999999e-05 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894117+0000", 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9204999999999999e-05 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894140+0000", 2024-09-16T04:08:23.441 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2719000000000001e-05 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.499011841000002, 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.442 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.443 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.444 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.447 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.498957780000001, 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:23.448 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:23.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:23.452 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.274689242000001, 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:23.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.457 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.696755282, 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.458 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:23.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s2 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.283981053, 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010866819, 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.462 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099684+0000", 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.004e-06 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099714+0000", 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9807000000000001e-05 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099769+0000", 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5306000000000002e-05 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099781+0000", 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2091e-05 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099835+0000", 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4058000000000003e-05 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099845+0000", 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5729999999999997e-06 2024-09-16T04:08:23.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110509+0000", 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010664339 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110550+0000", 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640999999999999e-05 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.279942137000001, 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:23.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:23.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:23.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.278554722999999, 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:23.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:23.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:23.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.276287668, 2024-09-16T04:08:23.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099172860000000009, 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107377+0000", 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8900000000000001e-07 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107437+0000", 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0628000000000003e-05 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107526+0000", 2024-09-16T04:08:23.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8873000000000001e-05 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107535+0000", 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0489999999999993e-06 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111543+0000", 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040080810000000001 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111554+0000", 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0321e-05 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117271+0000", 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057176529999999996 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117294+0000", 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2391999999999998e-05 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 42/27 ECSubWrite(tid=87, reqid=osd.5.0:87, at_version=42'39, trim_to=0'0, roll_forward_to=41'35))", 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.436384+0000", 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.9472797770000003, 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092481160000000007, 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436384+0000", 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436384+0000", 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436385+0000", 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2799999999999995e-07 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436405+0000", 2024-09-16T04:08:23.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0169e-05 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436443+0000", 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7920000000000003e-05 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436451+0000", 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4120000000000001e-06 2024-09-16T04:08:23.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436507+0000", 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5512e-05 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436516+0000", 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2699999999999993e-06 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444738+0000", 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082217950000000005 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.445632+0000", 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089431000000000005 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=188, reqid=client.4440.0:379, at_version=47'56, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:23.484 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.93142326600000003, 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011758437, 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452266+0000", 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4151e-05 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459768+0000", 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075025179999999997 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459778+0000", 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0244999999999999e-05 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459835+0000", 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6394999999999998e-05 2024-09-16T04:08:23.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459845+0000", 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0518e-05 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463963+0000", 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041182450000000004 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463999+0000", 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5667999999999998e-05 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.91923569299999996, 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.489 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:08:23.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:08:23.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:23.493 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:23.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:152: done (0 left) 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:153: rmattr oid 6 current snap is 9 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:153: done (0 left) 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:154: rmattr oid 44 current snap is 9 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:154: done (0 left) 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:155: setattr oid 37 current snap is 9 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:156: read oid 21 snap 8 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:156: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:157: rollback oid 47 current snap is 9 2024-09-16T04:08:23.496 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 47 to 8 2024-09-16T04:08:23.497 INFO:tasks.rados.rados.0.smithi016.stdout:158: snap_create 2024-09-16T04:08:23.497 INFO:tasks.rados.rados.0.smithi016.stdout:156: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-16T04:08:23.497 INFO:tasks.rados.rados.0.smithi016.stdout:157: finishing rollback tid 0 to smithi01639990-47 2024-09-16T04:08:23.497 INFO:tasks.rados.rados.0.smithi016.stdout:157: finishing rollback tid 1 to smithi01639990-47 2024-09-16T04:08:23.497 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 37 v 54 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T04:08:23.497 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:23.493+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:23.498 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:23.596 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:23.594+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:23.607 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:23.607 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:23.607 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:23.607 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:23.616 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:23.696 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:23.695+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:23.720 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:23.720 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:23.720 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:23.720 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:23.721 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:23.729 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:23.798 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:23.796+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:23.873 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:186 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~589824] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.194343+0000", 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.676842390999999, 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060370950000000001, 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.874 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 186 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.194343+0000", 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.194343+0000", 2024-09-16T04:08:23.875 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.194345+0000", 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.759e-06 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195173+0000", 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00082836499999999996 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.876 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195190+0000", 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6756999999999999e-05 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195198+0000", 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2599999999999999e-06 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195250+0000", 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1996999999999997e-05 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.195323+0000", 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3161e-05 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.196249+0000", 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00092679299999999997 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200145+0000", 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038957269999999999 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200279+0000", 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013368599999999999 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200380+0000", 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010158999999999999 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.879 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:189 3.9s0 3:92dcccbe:::smithi01639990-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:23.880 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.197539+0000", 2024-09-16T04:08:23.880 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.673646642, 2024-09-16T04:08:23.880 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005956355, 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 189 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.881 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197539+0000", 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197539+0000", 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197541+0000", 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0770000000000002e-06 2024-09-16T04:08:23.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197543+0000", 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1859999999999999e-06 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197550+0000", 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7020000000000004e-06 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.197555+0000", 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7809999999999996e-06 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200575+0000", 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030196649999999999 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.200666+0000", 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1508999999999994e-05 2024-09-16T04:08:23.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202294+0000", 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016276719999999999 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.202368+0000", 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4441000000000001e-05 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.203495+0000", 2024-09-16T04:08:23.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001127322 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(osd.11.18:1 3.9s0 3:92dcccbe:::smithi01639990-16:head [assert-version v23,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.214792+0000", 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.656393725999999, 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063663399999999998, 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214792+0000", 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214792+0000", 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214794+0000", 2024-09-16T04:08:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9640000000000002e-06 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214802+0000", 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9699999999999999e-06 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214822+0000", 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9737000000000001e-05 2024-09-16T04:08:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214828+0000", 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0159999999999997e-06 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214901+0000", 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3320999999999998e-05 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.214983+0000", 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2216e-05 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.221158+0000", 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061751159999999996 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.767692283000001, 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073296819999999997, 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103494+0000", 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1399999999999997e-07 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103514+0000", 2024-09-16T04:08:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9698999999999999e-05 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103592+0000", 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8378000000000001e-05 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103598+0000", 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9100000000000002e-06 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105790+0000", 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021916359999999998 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105798+0000", 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2490000000000001e-06 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110788+0000", 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049902660000000001 2024-09-16T04:08:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110823+0000", 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5129999999999997e-05 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=34, reqid=client.4440.0:211, at_version=33'20, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.755475347000001, 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078650630000000003, 2024-09-16T04:08:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115711+0000", 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2399999999999998e-07 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115771+0000", 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9478999999999998e-05 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119749+0000", 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039784099999999999 2024-09-16T04:08:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119760+0000", 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0664e-05 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119899+0000", 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013905300000000001 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119919+0000", 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9854999999999999e-05 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123544+0000", 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036256499999999998 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123575+0000", 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1328000000000002e-05 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=35, reqid=client.4440.0:212, at_version=33'21, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.752720225000001, 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077855709999999998, 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:08:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118551+0000", 2024-09-16T04:08:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4878999999999998e-05 2024-09-16T04:08:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120595+0000", 2024-09-16T04:08:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020432639999999999 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120604+0000", 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0250000000000008e-06 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122166+0000", 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015625599999999999 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122175+0000", 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8470000000000002e-06 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126217+0000", 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040415249999999998 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126251+0000", 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4570999999999998e-05 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.729492462, 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 37/27 ECSubWrite(tid=128, reqid=osd.3.0:124, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.111259+0000", 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.759926703, 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054317890000000002, 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111259+0000", 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111259+0000", 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111260+0000", 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5899999999999996e-07 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111269+0000", 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0950000000000002e-06 2024-09-16T04:08:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111299+0000", 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0320000000000001e-05 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111308+0000", 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7430000000000008e-06 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111360+0000", 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1959000000000002e-05 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.111369+0000", 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0829999999999993e-06 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116129+0000", 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047604600000000002 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116691+0000", 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00056147 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:296 3.9s0 3:9fd10cc9:::smithi01639990-50:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.7666125719999997, 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010626313, 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104575+0000", 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1349999999999999e-06 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104586+0000", 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0603e-05 2024-09-16T04:08:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104606+0000", 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9840999999999999e-05 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104615+0000", 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0189999999999995e-06 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104674+0000", 2024-09-16T04:08:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9534999999999998e-05 2024-09-16T04:08:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104680+0000", 2024-09-16T04:08:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9320000000000001e-06 2024-09-16T04:08:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109134+0000", 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044534120000000003 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109195+0000", 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0983000000000002e-05 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109711+0000", 2024-09-16T04:08:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051621600000000001 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.110753+0000", 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001042533 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115115+0000", 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043613030000000004 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115200+0000", 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4801000000000003e-05 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 42/27 ECSubWrite(tid=154, reqid=osd.3.0:154, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439636+0000", 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.4315500989999999, 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062793340000000001, 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439636+0000", 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439636+0000", 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439636+0000", 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8599999999999999e-07 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439650+0000", 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3954000000000001e-05 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440315+0000", 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066470199999999998 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440324+0000", 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5789999999999993e-06 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440849+0000", 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052441 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440859+0000", 2024-09-16T04:08:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0713000000000001e-05 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.445872+0000", 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050127039999999998 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.445915+0000", 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2985999999999998e-05 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443622+0000", 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.4275637950000002, 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0071718909999999997, 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443622+0000", 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443622+0000", 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443623+0000", 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.037e-06 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443939+0000", 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00031647100000000003 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447492+0000", 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0035522990000000001 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447506+0000", 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4268e-05 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447577+0000", 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0769e-05 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.447587+0000", 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0406e-05 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450760+0000", 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031730220000000002 2024-09-16T04:08:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450794+0000", 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3618999999999998e-05 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.422163201, 2024-09-16T04:08:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085197069999999996, 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4000000000000001e-07 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449160+0000", 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013637999999999999 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451237+0000", 2024-09-16T04:08:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020768179999999998 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451246+0000", 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451532+0000", 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00028571800000000002 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451542+0000", 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0118999999999999e-05 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457500+0000", 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059576680000000002 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457542+0000", 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2716999999999998e-05 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=160, reqid=client.4440.0:347, at_version=42'32, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.451166+0000", 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.4200198720000001, 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064616450000000002, 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451166+0000", 2024-09-16T04:08:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451166+0000", 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451167+0000", 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0809999999999999e-06 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451210+0000", 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3350000000000003e-05 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452031+0000", 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00082036499999999998 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452040+0000", 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7890000000000006e-06 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452121+0000", 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0725999999999999e-05 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452148+0000", 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6863999999999999e-05 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457572+0000", 2024-09-16T04:08:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054240030000000002 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457628+0000", 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5467000000000003e-05 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=161, reqid=client.4440.0:348, at_version=42'33, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.451985+0000", 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.4192009060000004, 2024-09-16T04:08:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056726499999999996, 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451985+0000", 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451985+0000", 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451986+0000", 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8899999999999998e-07 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452006+0000", 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0486999999999999e-05 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452781+0000", 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00077455899999999997 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452790+0000", 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.397e-06 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452896+0000", 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000106106 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.452906+0000", 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4069999999999999e-06 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457631+0000", 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047248689999999996 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457658+0000", 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6835999999999999e-05 2024-09-16T04:08:23.943 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.4114368539999997, 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.944 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:08:23.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:08:23.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:23.948 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.2808946949999997, 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:23.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.952 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:363 3.9s0 3:94de6fe1:::smithi01639990-3:head [stat,delete] snapc 8=[6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.4313280349999999, 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085012569999999999, 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:23.953 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 363 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.954 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439860+0000", 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439869+0000", 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3170000000000008e-06 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439888+0000", 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9293e-05 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439897+0000", 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439953+0000", 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6317999999999998e-05 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439959+0000", 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3040000000000004e-06 2024-09-16T04:08:23.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441596+0000", 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001637655 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441659+0000", 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2658999999999998e-05 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.442297+0000", 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063811800000000002 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.443562+0000", 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001265361 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448304+0000", 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047413560000000004 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448359+0000", 2024-09-16T04:08:23.959 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5189000000000003e-05 2024-09-16T04:08:23.959 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.959 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.959 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.959 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.959 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.959 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:08:23.960 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:23.960 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.199972142, 2024-09-16T04:08:23.960 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:08:23.960 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.960 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.960 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:08:23.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:23.964 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0395680839999999, 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:23.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.967 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:23.968 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.471436+0000", 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.39975026400000002, 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0071008989999999999, 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471436+0000", 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:23.969 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471436+0000", 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471436+0000", 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8500000000000002e-07 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471522+0000", 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6056000000000004e-05 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471964+0000", 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000442282 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471974+0000", 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9159999999999996e-06 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.473222+0000", 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001247827 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.473233+0000", 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0906e-05 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478496+0000", 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0052629729999999998 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478536+0000", 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0553999999999998e-05 2024-09-16T04:08:23.972 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.973 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.973 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.973 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:23.973 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:23.973 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:23.974 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:23.975 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:23.973+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:23.984 INFO:tasks.rados.rados.0.smithi016.stdout:157: finishing rollback tid 2 to smithi01639990-47 2024-09-16T04:08:23.984 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 47 v 38 (ObjNum 60 snap 2 seq_num 60) dirty exists 2024-09-16T04:08:24.075 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.073+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:24.078 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:24.078 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:24.078 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:24.078 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:24.087 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:24.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:24.176 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.174+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:24.191 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:24.191 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:24.191 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:24.191 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:24.191 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:24.200 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:24.259 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:24.259 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:24.277 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:24.274+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:24.280 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:24.280 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:24.301 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:24.301 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:24.304 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:24.304 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:24.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:24.304 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:24.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.304 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:24.304 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.207476767999999, 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:24.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es5 32/27 ECSubWrite(tid=28, reqid=client.4440.0:197, at_version=32'4, trim_to=0'0, roll_forward_to=30'2))", 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.227608+0000", 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.074176210000001, 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004862318, 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.309 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227608+0000", 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227608+0000", 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227610+0000", 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8369999999999999e-06 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228034+0000", 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042372999999999998 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228094+0000", 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0418000000000003e-05 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228107+0000", 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3121e-05 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228200+0000", 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2559000000000006e-05 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.228213+0000", 2024-09-16T04:08:24.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3574000000000001e-05 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232430+0000", 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004217213 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232470+0000", 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9866000000000001e-05 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.313 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.202036294999999, 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067064200000000003, 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099749+0000", 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1299999999999999e-07 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099771+0000", 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2642000000000001e-05 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099811+0000", 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9406999999999997e-05 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099819+0000", 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:24.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100401+0000", 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058156199999999999 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100411+0000", 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7079999999999992e-06 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106408+0000", 2024-09-16T04:08:24.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059972369999999999 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106454+0000", 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6384000000000002e-05 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103167+0000", 2024-09-16T04:08:24.318 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.198617229, 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0055620829999999998, 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103167+0000", 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103167+0000", 2024-09-16T04:08:24.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103175+0000", 2024-09-16T04:08:24.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2879999999999993e-06 2024-09-16T04:08:24.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103717+0000", 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00054184999999999997 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103749+0000", 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2314999999999998e-05 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103757+0000", 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2729999999999997e-06 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103808+0000", 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1072999999999997e-05 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103817+0000", 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8410000000000006e-06 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108704+0000", 2024-09-16T04:08:24.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048869250000000003 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108729+0000", 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5517999999999999e-05 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103841+0000", 2024-09-16T04:08:24.323 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.197943108, 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048534260000000001, 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103841+0000", 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103841+0000", 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103842+0000", 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4700000000000001e-07 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103856+0000", 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4748e-05 2024-09-16T04:08:24.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103886+0000", 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9437999999999998e-05 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103894+0000", 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9189999999999999e-06 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103978+0000", 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3775000000000003e-05 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103983+0000", 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.519e-06 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108661+0000", 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004677858 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.108694+0000", 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3522000000000002e-05 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.197589383, 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062253660000000004, 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.328 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104196+0000", 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104218+0000", 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1974999999999999e-05 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104245+0000", 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7489999999999999e-05 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104254+0000", 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104313+0000", 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9151000000000003e-05 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104322+0000", 2024-09-16T04:08:24.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110389+0000", 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006067205 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110420+0000", 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1100999999999999e-05 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.332 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.106795+0000", 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.194988707, 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056249589999999997, 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106795+0000", 2024-09-16T04:08:24.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106795+0000", 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106796+0000", 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.79e-07 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106819+0000", 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3085999999999999e-05 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106897+0000", 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8067999999999999e-05 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106905+0000", 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2209999999999997e-06 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106978+0000", 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2822000000000002e-05 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.106988+0000", 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0468999999999999e-05 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112382+0000", 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005393424 2024-09-16T04:08:24.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112420+0000", 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8689999999999997e-05 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=163, reqid=osd.7.0:163, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.191810096999999, 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053772749999999999, 2024-09-16T04:08:24.337 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.109974+0000", 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1799999999999999e-07 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110009+0000", 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4467000000000001e-05 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110085+0000", 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6359999999999994e-05 2024-09-16T04:08:24.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110093+0000", 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0879999999999995e-06 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110482+0000", 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00038852100000000002 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110498+0000", 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5858999999999998e-05 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115317+0000", 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0048192579999999999 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.115351+0000", 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4504e-05 2024-09-16T04:08:24.341 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.191772861, 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062070839999999999, 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110012+0000", 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3999999999999997e-07 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110025+0000", 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3053e-05 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110784+0000", 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075944100000000002 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110793+0000", 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4880000000000007e-06 2024-09-16T04:08:24.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110853+0000", 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0333e-05 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110861+0000", 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2090000000000005e-06 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116184+0000", 2024-09-16T04:08:24.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053227389999999999 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116218+0000", 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4480999999999997e-05 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 37/27 ECSubWrite(tid=129, reqid=osd.3.0:129, at_version=37'43, trim_to=0'0, roll_forward_to=35'41))", 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.189007074999999, 2024-09-16T04:08:24.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061976449999999999, 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112778+0000", 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3499999999999996e-07 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112797+0000", 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9578000000000001e-05 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112842+0000", 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5093999999999999e-05 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112850+0000", 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112972+0000", 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000122033 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112982+0000", 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5789999999999993e-06 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118944+0000", 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005961816 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118975+0000", 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0933999999999999e-05 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.350 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=155, reqid=osd.3.0:155, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.8621439620000002, 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073085160000000001, 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1299999999999999e-07 2024-09-16T04:08:24.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439663+0000", 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3030000000000001e-05 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439705+0000", 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1078999999999998e-05 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439714+0000", 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441461+0000", 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017466350000000001 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441475+0000", 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4404e-05 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446911+0000", 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054361430000000001 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446949+0000", 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7465000000000001e-05 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:24.355 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.858065195, 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068399539999999997, 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443720+0000", 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9500000000000001e-07 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444146+0000", 2024-09-16T04:08:24.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042652899999999998 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444178+0000", 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1412999999999999e-05 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444185+0000", 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4270000000000002e-06 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444247+0000", 2024-09-16T04:08:24.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1760000000000005e-05 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444254+0000", 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9260000000000003e-06 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450519+0000", 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062649960000000001 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450559+0000", 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0108000000000003e-05 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s5 42/27 ECSubWrite(tid=40, reqid=osd.0.0:40, at_version=42'11, trim_to=0'0, roll_forward_to=41'9))", 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.8578911869999999, 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067397389999999998, 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.360 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0700000000000001e-07 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443914+0000", 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1274e-05 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443937+0000", 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2991999999999999e-05 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443945+0000", 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4780000000000002e-06 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444021+0000", 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5814000000000003e-05 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444033+0000", 2024-09-16T04:08:24.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2322e-05 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450586+0000", 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065529869999999997 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450633+0000", 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6764999999999997e-05 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.364 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.8719597979999998, 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:08:24.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:08:24.368 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.4120095710000005, 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:24.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:24.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:24.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:08:24.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:08:24.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.376 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.464615+0000", 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.83716886, 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0052095969999999998, 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.464615+0000", 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.464615+0000", 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.464616+0000", 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2500000000000004e-07 2024-09-16T04:08:24.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.464629+0000", 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2422000000000001e-05 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.464666+0000", 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7289000000000002e-05 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.464676+0000", 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.024e-05 2024-09-16T04:08:24.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.464744+0000", 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7612000000000003e-05 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.464754+0000", 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8679999999999994e-06 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.469777+0000", 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050231549999999996 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.469825+0000", 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8186000000000002e-05 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0884969659999999, 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:24.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:08:24.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:08:24.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:24.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:08:24.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:08:24.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:08:24.386 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:08:24.386 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.386 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.386 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.386 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 48/46 ECSubWrite(tid=190, reqid=osd.3.0:190, at_version=48'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:24.386 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.465574+0000", 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.83621054299999997, 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047946380000000004, 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465574+0000", 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465574+0000", 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465575+0000", 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9299999999999996e-07 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465620+0000", 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5506000000000001e-05 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465658+0000", 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465669+0000", 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1569000000000001e-05 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465784+0000", 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011468000000000001 2024-09-16T04:08:24.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465797+0000", 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2689e-05 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.470334+0000", 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0045372620000000002 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.470368+0000", 2024-09-16T04:08:24.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4230999999999998e-05 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.83071310799999998, 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078488229999999996, 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471072+0000", 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0279999999999999e-06 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471418+0000", 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034541800000000001 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471457+0000", 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471492+0000", 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:08:24.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471798+0000", 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030615599999999997 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471812+0000", 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.448e-05 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478881+0000", 2024-09-16T04:08:24.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070685549999999998 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478920+0000", 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9072000000000001e-05 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 48/46 ECSubWrite(tid=195, reqid=osd.3.0:190, at_version=0'0, trim_to=0'0, roll_forward_to=48'58))", 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.476769+0000", 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.82501538500000005, 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050717219999999999, 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476769+0000", 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476769+0000", 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476770+0000", 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5199999999999996e-07 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476782+0000", 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2189e-05 2024-09-16T04:08:24.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476823+0000", 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1523999999999997e-05 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476841+0000", 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8016000000000002e-05 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476909+0000", 2024-09-16T04:08:24.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8165999999999998e-05 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476921+0000", 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1089e-05 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.481789+0000", 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004868944 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.481841+0000", 2024-09-16T04:08:24.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1041999999999997e-05 2024-09-16T04:08:24.400 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.400 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.400 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.400 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:24.400 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:24.400 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:24.401 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:24.402 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:24.403 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:24.404 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:24.404 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:24.405 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:24.399+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:24.409 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:24.429 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:24.450 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:24.459 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T04:08:24.460 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:08:24.460 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-16T04:08:24.460 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T04:08:24.460 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T04:08:24.460 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-16T04:08:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-16T04:08:24.467 INFO:tasks.rados.rados.0.smithi016.stdout:155: done (3 left) 2024-09-16T04:08:24.467 INFO:tasks.rados.rados.0.smithi016.stdout:156: done (2 left) 2024-09-16T04:08:24.467 INFO:tasks.rados.rados.0.smithi016.stdout:157: done (1 left) 2024-09-16T04:08:24.467 INFO:tasks.rados.rados.0.smithi016.stdout:158: done (0 left) 2024-09-16T04:08:24.467 INFO:tasks.rados.rados.0.smithi016.stdout:159: rmattr oid 33 current snap is 10 2024-09-16T04:08:24.467 INFO:tasks.rados.rados.0.smithi016.stdout:159: done (0 left) 2024-09-16T04:08:24.467 INFO:tasks.rados.rados.0.smithi016.stdout:160: snap_remove snap 8 2024-09-16T04:08:24.471 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.465+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.471 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.465+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.471 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.465+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.471 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.471 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.472 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.473 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.474 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.466+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.475 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.476 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.477 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.477 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.477 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.477 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.477 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.467+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.480 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.481 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.467+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.482 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.483 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.484 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.485 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.485 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.485 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.485 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.485 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.485 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.485 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.468+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.486 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.487 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.488 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.489 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.489 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.489 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.489 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.489 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.489 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f650c917640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.490 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.491 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.491 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.491 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.491 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.491 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.491 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.492 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.493 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.494 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.495 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.496 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.498 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.498 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.498 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.498 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.498 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.498 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.500 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.500 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.500 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.500 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.500 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.500 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.500 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.500 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.500 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.501 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.501 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.501 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.501 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.501 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.501 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.501 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.501 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.501 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.502 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.503 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.470+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.504 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.505 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.506 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.509 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.510 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.511 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.512 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.470+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.512 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.513 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.513 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.513 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.513 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.474+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.515 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.516 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.517 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.518 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.519 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.520 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.520 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.520 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.520 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.520 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.520 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.520 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.520 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.521 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.521 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.521 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.521 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.471+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.521 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.521 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.521 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.521 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.522 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.522 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.522 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.522 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.522 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.522 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.522 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.522 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.522 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.475+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.523 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.528 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.529 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.468+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.530 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.531 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.532 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.532 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.532 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.532 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.532 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.532 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.532 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.532 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.533 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.470+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.534 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.535 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.536 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.537 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.538 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.539 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.540 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.471+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.541 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.542 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.543 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.544 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.472+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.545 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.476+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.546 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.547 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.548 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.548 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.549 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.549 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.549 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.550 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.550 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.550 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.550 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.551 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.551 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.551 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.552 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.552 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.552 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.552 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.552 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.552 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.552 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.552 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.554 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.554 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.554 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.554 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.554 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.554 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.554 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.554 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.554 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.556 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.559 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.560 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.561 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.562 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.563 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.564 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.564 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.564 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.564 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.564 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.564 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.564 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.476+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.564 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.565 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.473+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.569 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.569 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.569 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.569 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.569 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.570 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.571 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.571 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.571 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.571 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.571 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.571 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.571 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.571 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.572 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.573 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.574 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.575 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.575 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.575 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.575 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.575 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.575 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.576 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.576 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.576 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.576 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.576 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.576 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.477+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.577 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.578 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.579 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.580 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.581 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.581 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.582 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.583 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.584 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.584 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.584 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.584 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.584 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.584 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.584 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.584 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.584 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.585 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.586 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.586 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.586 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.586 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.586 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.586 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.586 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.586 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.586 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.587 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.587 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.588 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.589 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.469+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.470+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.590 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.591 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.591 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.591 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.591 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.473+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.592 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.592 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.592 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.593 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.473+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.595 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.596 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.596 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.596 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.471+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.596 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.473+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.597 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.597 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.473+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.599 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.599 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.599 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.599 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.600 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.600 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.602 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.602 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.602 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.602 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.602 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.602 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.602 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.603 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.603 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.603 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.603 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.603 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.604 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.604 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.604 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.604 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.605 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.605 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.605 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.472+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.611 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.473+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.612 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.613 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.613 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.613 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.613 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.615 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.615 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.615 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.615 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.615 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.615 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.615 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.615 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.616 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.616 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.616 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.616 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.616 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.616 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.616 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.616 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.616 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.478+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.617 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.618 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.619 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.620 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.621 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.622 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.623 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.623 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.624 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.624 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.625 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.478+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.625 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.479+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.625 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.625 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.625 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.626 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.626 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.626 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.626 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.627 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.627 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.627 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.627 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.627 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.627 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.627 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.479+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.628 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.479+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.628 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.479+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.628 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.479+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.628 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.628 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.628 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.629 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.474+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.476+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.476+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.476+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.476+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.476+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.476+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.476+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.630 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.631 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.631 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.631 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.631 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.631 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.631 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.631 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.631 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.479+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.632 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.479+0000 7f3054e99640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.632 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.632 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.632 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.632 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.632 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.633 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.634 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.472+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.635 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.636 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.636 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.474+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.637 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.637 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.637 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.637 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.637 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.637 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.637 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.637 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.638 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.638 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.638 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.638 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.639 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.639 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.639 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.639 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.640 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.641 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.641 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.641 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.641 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.642 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.642 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.642 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.643 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.643 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.644 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.644 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.644 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.644 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.645 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.645 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.646 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.646 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.646 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.647 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.647 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.647 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.647 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.479+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.649 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.649 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.649 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.649 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.650 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.651 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.476+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.652 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.653 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.654 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.654 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.654 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.654 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.654 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.654 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.654 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.654 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.477+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.655 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.656 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.657 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.478+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.658 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.659 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.660 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.660 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.660 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.660 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.660 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.660 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.661 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.661 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.661 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.661 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:24.479+0000 7f37c41d9640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,5,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.662 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.663 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.663 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.664 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:24.664 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:24.664 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:24.664 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:24.665 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:24.665 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.594+0000 7f9d5389b640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.666 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.666 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.666 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.666 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.666 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.668 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.669 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.669 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.669 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.669 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.669 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.669 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.669 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2ff208e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.670 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.670 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.670 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.670 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.670 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.671 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.671 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.671 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.671 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.671 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.672 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.674 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.674 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.674 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.674 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.674 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.675 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.676 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.676 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.677 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.677 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.677 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.678 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.678 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.679 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.679 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.680 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.680 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.680 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.680 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.681 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.681 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.681 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.480+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.682 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.682 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.682 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.683 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.683 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.683 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.683 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.683 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.684 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.685 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.685 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.685 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.686 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.686 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.686 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.686 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.686 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.687 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.687 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.687 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.687 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.687 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.688 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.688 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.689 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.689 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.689 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.689 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.689 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.690 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.690 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.690 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.690 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.690 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.690 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.691 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.691 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.692 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.692 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.692 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.692 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.692 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.693 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.693 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.693 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.693 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.694 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.694 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.694 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.694 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.695 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.695 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.695 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.696 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.696 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.696 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.696 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.481+0000 7f82b82c4640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.707 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.708 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.709 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.710 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.482+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.711 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.712 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.714 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.483+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,2,4,5,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.510+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.715 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.511+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:24.716 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.521+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.716 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:24.524+0000 7f82b42bc640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,4 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.717 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:24.540+0000 7f2fee086640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.717 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:24.602+0000 7f651091f640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,6 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.718 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.718 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.718 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.478+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.479+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.721 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.721 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.721 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.721 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.721 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.721 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.721 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.721 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.722 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.723 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.724 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.480+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.729 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.482+0000 7ffa766ed640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.734 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.483+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.737 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.738 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.739 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.741 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.484+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.745 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.746 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.747 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.485+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.749 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.751 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.752 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.753 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.486+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.489+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.758 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.490+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.760 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.761 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.762 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.491+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.766 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.767 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.767 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.767 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.767 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.492+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 2,3,6,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.768 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:24.507+0000 7f3050e91640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:24.769 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:08:24.769 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T04:08:24.769 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:08:24.769 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:24.622+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:24.770 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:24.770 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:24.770 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:24.771 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:24.771 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:24.768+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:24.775 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:24.775 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:24.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:24.871 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:24.869+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:24.886 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:24.886 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:24.895 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:24.979 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:24.977+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:25.039 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:25.039 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:25.048 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:25.069 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:25.067+0000 7ffa726e5640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:25.072 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:25.070+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:25.173 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:25.171+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:25.193 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:25.193 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:25.201 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:25.219 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:25.217+0000 7f9d578a3640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:25.274 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:25.272+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:25.283 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:25.281+0000 7f37c01d1640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:25.324 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:25.324 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:25.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:25.377 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:25.375+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:25.399 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:25.397+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:25.446 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:25.446 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:25.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:25.468 INFO:tasks.rados.rados.0.smithi016.stdout:160: done (0 left) 2024-09-16T04:08:25.468 INFO:tasks.rados.rados.0.smithi016.stdout:161: delete oid 39 current snap is 10 2024-09-16T04:08:25.475 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:25.473+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:25.476 INFO:tasks.rados.rados.0.smithi016.stdout:161: done (0 left) 2024-09-16T04:08:25.477 INFO:tasks.rados.rados.0.smithi016.stdout:162: snap_create 2024-09-16T04:08:25.515 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:25.513+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:25.547 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:25.545+0000 7ffa78ef2640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:25.569 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:25.567+0000 7ffa74eea640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:25.576 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:25.574+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:25.596 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:25.597 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:25.603 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:25.678 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:25.676+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:25.751 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:25.760 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:08:25.805 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:25.803+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:25.867 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:25.867 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:25.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:08:25.879 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:25.877+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:25.980 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:25.978+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:26.026 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:26.026 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:26.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:08:26.080 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:26.079+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:26.180 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:26.179+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:26.182 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:26.182 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:26.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:26.281 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:26.279+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:26.298 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:26.298 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:26.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:26.383 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:26.381+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:26.413 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:26.413 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:26.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:26.472 INFO:tasks.rados.rados.0.smithi016.stdout:162: done (0 left) 2024-09-16T04:08:26.472 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 45 current snap is 11 2024-09-16T04:08:26.473 INFO:tasks.rados.rados.0.smithi016.stdout:163: seq_num 86 ranges {278528=688128,966656=540672,1507328=229376} 2024-09-16T04:08:26.477 INFO:tasks.rados.rados.0.smithi016.stdout:163: writing smithi01639990-45 from 278528 to 966656 tid 1 2024-09-16T04:08:26.480 INFO:tasks.rados.rados.0.smithi016.stdout:163: writing smithi01639990-45 from 966656 to 1507328 tid 2 2024-09-16T04:08:26.483 INFO:tasks.rados.rados.0.smithi016.stdout:163: writing smithi01639990-45 from 1507328 to 1736704 tid 3 2024-09-16T04:08:26.483 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 10 current snap is 11 2024-09-16T04:08:26.483 INFO:tasks.rados.rados.0.smithi016.stdout:164: seq_num 87 ranges {0=704512,704512=688128,1392640=81920} 2024-09-16T04:08:26.483 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:26.480+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:26.486 INFO:tasks.rados.rados.0.smithi016.stdout:164: writing smithi01639990-10 from 0 to 704512 tid 1 2024-09-16T04:08:26.490 INFO:tasks.rados.rados.0.smithi016.stdout:164: writing smithi01639990-10 from 704512 to 1392640 tid 2 2024-09-16T04:08:26.491 INFO:tasks.rados.rados.0.smithi016.stdout:164: writing smithi01639990-10 from 1392640 to 1474560 tid 3 2024-09-16T04:08:26.491 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 49 current snap is 11 2024-09-16T04:08:26.491 INFO:tasks.rados.rados.0.smithi016.stdout:163: finishing write tid 1 to smithi01639990-45 2024-09-16T04:08:26.491 INFO:tasks.rados.rados.0.smithi016.stdout:165: seq_num 88 ranges {819200=720896,1540096=442368,1982464=327680} 2024-09-16T04:08:26.499 INFO:tasks.rados.rados.0.smithi016.stdout:165: writing smithi01639990-49 from 819200 to 1540096 tid 1 2024-09-16T04:08:26.502 INFO:tasks.rados.rados.0.smithi016.stdout:165: writing smithi01639990-49 from 1540096 to 1982464 tid 2 2024-09-16T04:08:26.504 INFO:tasks.rados.rados.0.smithi016.stdout:165: writing smithi01639990-49 from 1982464 to 2310144 tid 3 2024-09-16T04:08:26.504 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 3 current snap is 11 2024-09-16T04:08:26.504 INFO:tasks.rados.rados.0.smithi016.stdout:166: seq_num 89 ranges {0=737280,737280=573440,1310720=196608} 2024-09-16T04:08:26.509 INFO:tasks.rados.rados.0.smithi016.stdout:166: writing smithi01639990-3 from 0 to 737280 tid 1 2024-09-16T04:08:26.512 INFO:tasks.rados.rados.0.smithi016.stdout:166: writing smithi01639990-3 from 737280 to 1310720 tid 2 2024-09-16T04:08:26.514 INFO:tasks.rados.rados.0.smithi016.stdout:166: writing smithi01639990-3 from 1310720 to 1507328 tid 3 2024-09-16T04:08:26.515 INFO:tasks.rados.rados.0.smithi016.stdout:167: delete oid 48 current snap is 11 2024-09-16T04:08:26.515 INFO:tasks.rados.rados.0.smithi016.stdout:163: finishing write tid 2 to smithi01639990-45 2024-09-16T04:08:26.515 INFO:tasks.rados.rados.0.smithi016.stdout:163: finishing write tid 3 to smithi01639990-45 2024-09-16T04:08:26.515 INFO:tasks.rados.rados.0.smithi016.stdout:164: finishing write tid 1 to smithi01639990-10 2024-09-16T04:08:26.515 INFO:tasks.rados.rados.0.smithi016.stdout:164: finishing write tid 2 to smithi01639990-10 2024-09-16T04:08:26.515 INFO:tasks.rados.rados.0.smithi016.stdout:164: finishing write tid 3 to smithi01639990-10 2024-09-16T04:08:26.515 INFO:tasks.rados.rados.0.smithi016.stdout:165: finishing write tid 1 to smithi01639990-49 2024-09-16T04:08:26.516 INFO:tasks.rados.rados.0.smithi016.stdout:165: finishing write tid 2 to smithi01639990-49 2024-09-16T04:08:26.519 INFO:tasks.rados.rados.0.smithi016.stdout:166: finishing write tid 1 to smithi01639990-3 2024-09-16T04:08:26.519 INFO:tasks.rados.rados.0.smithi016.stdout:167: done (4 left) 2024-09-16T04:08:26.520 INFO:tasks.rados.rados.0.smithi016.stdout:168: snap_create 2024-09-16T04:08:26.523 INFO:tasks.rados.rados.0.smithi016.stdout:166: finishing write tid 2 to smithi01639990-3 2024-09-16T04:08:26.523 INFO:tasks.rados.rados.0.smithi016.stdout:166: finishing write tid 3 to smithi01639990-3 2024-09-16T04:08:26.523 INFO:tasks.rados.rados.0.smithi016.stdout:165: finishing write tid 3 to smithi01639990-49 2024-09-16T04:08:26.523 INFO:tasks.rados.rados.0.smithi016.stdout:165: finishing write tid 4 to smithi01639990-49 2024-09-16T04:08:26.524 INFO:tasks.rados.rados.0.smithi016.stdout:165: finishing write tid 5 to smithi01639990-49 2024-09-16T04:08:26.524 INFO:tasks.rados.rados.0.smithi016.stdout:165: oid 49 updating version 0 to 33 2024-09-16T04:08:26.525 INFO:tasks.rados.rados.0.smithi016.stdout:165: oid 49 updating version 33 to 35 2024-09-16T04:08:26.525 INFO:tasks.rados.rados.0.smithi016.stdout:165: oid 49 version 35 is already newer than 34 2024-09-16T04:08:26.525 INFO:tasks.rados.rados.0.smithi016.stdout:165: oid 49 version 35 is already newer than 32 2024-09-16T04:08:26.525 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 49 v 35 (ObjNum 88 snap 11 seq_num 88) dirty exists 2024-09-16T04:08:26.525 INFO:tasks.rados.rados.0.smithi016.stdout:165: left oid 49 (ObjNum 88 snap 11 seq_num 88) 2024-09-16T04:08:26.527 INFO:tasks.rados.rados.0.smithi016.stdout:166: finishing write tid 4 to smithi01639990-3 2024-09-16T04:08:26.529 INFO:tasks.rados.rados.0.smithi016.stdout:166: finishing write tid 5 to smithi01639990-3 2024-09-16T04:08:26.529 INFO:tasks.rados.rados.0.smithi016.stdout:166: oid 3 updating version 0 to 42 2024-09-16T04:08:26.529 INFO:tasks.rados.rados.0.smithi016.stdout:166: oid 3 updating version 42 to 43 2024-09-16T04:08:26.529 INFO:tasks.rados.rados.0.smithi016.stdout:166: oid 3 version 43 is already newer than 41 2024-09-16T04:08:26.530 INFO:tasks.rados.rados.0.smithi016.stdout:166: oid 3 version 43 is already newer than 40 2024-09-16T04:08:26.530 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 3 v 43 (ObjNum 89 snap 11 seq_num 89) dirty exists 2024-09-16T04:08:26.530 INFO:tasks.rados.rados.0.smithi016.stdout:166: left oid 3 (ObjNum 89 snap 11 seq_num 89) 2024-09-16T04:08:26.541 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:26.541 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:26.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:26.584 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:26.581+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:26.603 INFO:tasks.rados.rados.0.smithi016.stdout:164: finishing write tid 4 to smithi01639990-10 2024-09-16T04:08:26.605 INFO:tasks.rados.rados.0.smithi016.stdout:164: finishing write tid 5 to smithi01639990-10 2024-09-16T04:08:26.605 INFO:tasks.rados.rados.0.smithi016.stdout:164: oid 10 updating version 0 to 7 2024-09-16T04:08:26.605 INFO:tasks.rados.rados.0.smithi016.stdout:164: oid 10 updating version 7 to 8 2024-09-16T04:08:26.605 INFO:tasks.rados.rados.0.smithi016.stdout:164: oid 10 version 8 is already newer than 5 2024-09-16T04:08:26.605 INFO:tasks.rados.rados.0.smithi016.stdout:164: oid 10 version 8 is already newer than 6 2024-09-16T04:08:26.605 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 10 v 8 (ObjNum 87 snap 11 seq_num 87) dirty exists 2024-09-16T04:08:26.605 INFO:tasks.rados.rados.0.smithi016.stdout:164: left oid 10 (ObjNum 87 snap 11 seq_num 87) 2024-09-16T04:08:26.658 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:26.658 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:26.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:26.684 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:26.682+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:26.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:08:26.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:26.774 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:26.774 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:26.783 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:26.825 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:26.822+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:26.886 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:26.883+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:26.923 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:26.923 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:26.945 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:26.946 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:26.947 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:26.956 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:26.967 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:26.967 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:26.986 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:26.985+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:26.988 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:26.988 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:27.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:27.030 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:27.031 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:27.051 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:27.052 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:27.068 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:27.068 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:27.068 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:27.068 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:27.072 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:27.077 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:27.087 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:27.085+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:27.094 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:27.116 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:27.138 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:27.139 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:27.161 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:27.161 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:27.180 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:27.180 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:27.180 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:27.180 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:27.180 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:27.186 INFO:tasks.rados.rados.0.smithi016.stdout:163: finishing write tid 4 to smithi01639990-45 2024-09-16T04:08:27.187 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:27.185+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:27.188 INFO:tasks.rados.rados.0.smithi016.stdout:163: finishing write tid 5 to smithi01639990-45 2024-09-16T04:08:27.188 INFO:tasks.rados.rados.0.smithi016.stdout:163: oid 45 updating version 0 to 21 2024-09-16T04:08:27.188 INFO:tasks.rados.rados.0.smithi016.stdout:163: oid 45 updating version 21 to 22 2024-09-16T04:08:27.188 INFO:tasks.rados.rados.0.smithi016.stdout:163: oid 45 version 22 is already newer than 19 2024-09-16T04:08:27.189 INFO:tasks.rados.rados.0.smithi016.stdout:163: oid 45 version 22 is already newer than 20 2024-09-16T04:08:27.189 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 45 v 22 (ObjNum 86 snap 11 seq_num 86) dirty exists 2024-09-16T04:08:27.189 INFO:tasks.rados.rados.0.smithi016.stdout:163: left oid 45 (ObjNum 86 snap 11 seq_num 86) 2024-09-16T04:08:27.190 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:27.288 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:27.287+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:27.296 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:27.296 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:27.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:199 3.3s0 3:c05d4807:::smithi01639990-31:head [copy-from ver 11] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.083026233999998, 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.48371305599999997, 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 199 2024-09-16T04:08:27.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209959+0000", 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7460000000000001e-06 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209971+0000", 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2787e-05 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210000+0000", 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8952999999999999e-05 2024-09-16T04:08:27.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210015+0000", 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.434e-05 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210169+0000", 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015455199999999999 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210317+0000", 2024-09-16T04:08:27.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000147291 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.224854+0000", 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014536926 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226325+0000", 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001471099 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693612+0000", 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.467287324 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693669+0000", 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7037999999999998e-05 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.302 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:200 3.3s0 3:c05d4807:::smithi01639990-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.082935944999999, 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016139113, 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.303 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 200 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210047+0000", 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210051+0000", 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8800000000000001e-06 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210065+0000", 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3567000000000001e-05 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210074+0000", 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2399999999999996e-06 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210464+0000", 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00038978800000000002 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210496+0000", 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2539999999999997e-05 2024-09-16T04:08:27.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.225987+0000", 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015491398999999999 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226084+0000", 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6618000000000002e-05 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226185+0000", 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101205 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s6 33/27 ECSubWrite(tid=37, reqid=osd.8.0:37, at_version=33'25, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.098670+0000", 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.194312192000002, 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058761789999999996, 2024-09-16T04:08:27.308 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098670+0000", 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098670+0000", 2024-09-16T04:08:27.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098670+0000", 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2700000000000001e-07 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098688+0000", 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7703e-05 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098726+0000", 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7818000000000003e-05 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098735+0000", 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9109999999999999e-06 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098816+0000", 2024-09-16T04:08:27.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1668999999999993e-05 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098825+0000", 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2979999999999997e-06 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104517+0000", 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0056911380000000001 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104546+0000", 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9515000000000001e-05 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s6 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.193392006, 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010829642, 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.313 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099591+0000", 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4399999999999999e-07 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099619+0000", 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7555e-05 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099655+0000", 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6337000000000001e-05 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099691+0000", 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5994000000000001e-05 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099819+0000", 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000128484 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099829+0000", 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1949999999999992e-06 2024-09-16T04:08:27.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110366+0000", 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010537504 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110420+0000", 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3729e-05 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s2 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.191575373999999, 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012298224, 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000002e-07 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101422+0000", 2024-09-16T04:08:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5004000000000001e-05 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101446+0000", 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3632999999999999e-05 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101452+0000", 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0260000000000004e-06 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101511+0000", 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9515000000000002e-05 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101523+0000", 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1094000000000001e-05 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113680+0000", 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012157401 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113705+0000", 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5041999999999998e-05 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.es3 33/27 ECSubWrite(tid=31, reqid=osd.11.0:31, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-16T04:08:27.322 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.191457024000002, 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012148897000000001, 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101526+0000", 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0900000000000001e-07 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101543+0000", 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7215999999999999e-05 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101584+0000", 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1352e-05 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101590+0000", 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5779999999999998e-06 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101650+0000", 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0093000000000002e-05 2024-09-16T04:08:27.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101662+0000", 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2257e-05 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113631+0000", 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.01196856 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113674+0000", 2024-09-16T04:08:27.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3132e-05 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.191434008000002, 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012128856, 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.327 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:08:27.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2399999999999999e-07 2024-09-16T04:08:27.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101558+0000", 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1980000000000007e-06 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101618+0000", 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0911e-05 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101624+0000", 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6899999999999997e-06 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101690+0000", 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5988000000000006e-05 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101701+0000", 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0878e-05 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113645+0000", 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011944110000000001 2024-09-16T04:08:27.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113677+0000", 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1856999999999997e-05 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.189954954000001, 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014214867000000001, 2024-09-16T04:08:27.332 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103028+0000", 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8999999999999995e-07 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103051+0000", 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2909999999999999e-05 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103122+0000", 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1339000000000003e-05 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103130+0000", 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8790000000000003e-06 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104942+0000", 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001811893 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104952+0000", 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5449999999999992e-06 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117201+0000", 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012249395 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117242+0000", 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1016000000000002e-05 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.188427394000001, 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012207213999999999, 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.337 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104556+0000", 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104963+0000", 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000407604 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105023+0000", 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9684e-05 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105031+0000", 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3599999999999996e-06 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105239+0000", 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00020760900000000001 2024-09-16T04:08:27.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105251+0000", 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1788e-05 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116731+0000", 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011480292 2024-09-16T04:08:27.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116762+0000", 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1087000000000002e-05 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.107022+0000", 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.185960452, 2024-09-16T04:08:27.342 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097676329999999995, 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107022+0000", 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107022+0000", 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.343 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107022+0000", 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2900000000000003e-07 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107282+0000", 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025955999999999999 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107317+0000", 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5176999999999999e-05 2024-09-16T04:08:27.344 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107326+0000", 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1220000000000001e-06 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111509+0000", 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041823099999999998 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111518+0000", 2024-09-16T04:08:27.345 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4870000000000008e-06 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116766+0000", 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052478159999999998 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116789+0000", 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3431999999999999e-05 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.346 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=80, reqid=client.4440.0:209, at_version=33'23, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.108124+0000", 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.184858415000001, 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086904689999999993, 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108124+0000", 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.347 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108124+0000", 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108124+0000", 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.348 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108135+0000", 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1338000000000001e-05 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108191+0000", 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6217000000000001e-05 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.108197+0000", 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8819999999999998e-06 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.349 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111944+0000", 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037468250000000001 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111953+0000", 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7269999999999996e-06 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116794+0000", 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048407049999999998 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.350 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116814+0000", 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0602e-05 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=81, reqid=client.4440.0:202, at_version=33'25, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.113700+0000", 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.179282153999999, 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079098669999999992, 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.351 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113700+0000", 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113700+0000", 2024-09-16T04:08:27.352 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113701+0000", 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9299999999999997e-07 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113865+0000", 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016396499999999999 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.353 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113890+0000", 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5024e-05 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113897+0000", 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6090000000000003e-06 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.354 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113949+0000", 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2037999999999999e-05 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113957+0000", 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1940000000000001e-06 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121563+0000", 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0076067779999999998 2024-09-16T04:08:27.355 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121610+0000", 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6465999999999999e-05 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.es3 33/27 ECSubWrite(tid=33, reqid=osd.11.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=33'6))", 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.114128+0000", 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.178854164000001, 2024-09-16T04:08:27.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074696759999999997, 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114128+0000", 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114128+0000", 2024-09-16T04:08:27.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114129+0000", 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6000000000000003e-07 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114139+0000", 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0764999999999999e-05 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114164+0000", 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5001999999999999e-05 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114170+0000", 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9000000000000003e-06 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114215+0000", 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4308000000000003e-05 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114221+0000", 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0430000000000004e-06 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121555+0000", 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073344179999999997 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121598+0000", 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2580000000000002e-05 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.360 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s2 33/27 ECSubWrite(tid=17, reqid=osd.0.0:16, at_version=0'0, trim_to=0'0, roll_forward_to=33'8))", 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.114574+0000", 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.178407920000001, 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070510010000000003, 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.361 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114574+0000", 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114574+0000", 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114575+0000", 2024-09-16T04:08:27.362 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3200000000000003e-07 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114585+0000", 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9140000000000003e-06 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114619+0000", 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4289999999999999e-05 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114625+0000", 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8939999999999998e-06 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114671+0000", 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5636999999999997e-05 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.114681+0000", 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0298e-05 2024-09-16T04:08:27.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121602+0000", 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069206199999999997 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.121625+0000", 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3416000000000001e-05 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.365 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=34, reqid=client.4440.0:211, at_version=33'20, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.115642+0000", 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.177340274999999, 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0063991539999999998, 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.366 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115642+0000", 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115642+0000", 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115643+0000", 2024-09-16T04:08:27.367 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4200000000000005e-07 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115822+0000", 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000179273 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115863+0000", 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1133000000000001e-05 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115870+0000", 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115941+0000", 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0547000000000001e-05 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.115947+0000", 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3810000000000004e-06 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.122004+0000", 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060568369999999998 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.122041+0000", 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7049999999999999e-05 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s3 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.141650+0000", 2024-09-16T04:08:27.370 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.151332372999999, 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062685240000000001, 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141650+0000", 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141650+0000", 2024-09-16T04:08:27.371 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141651+0000", 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5099999999999999e-07 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141667+0000", 2024-09-16T04:08:27.372 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6458000000000001e-05 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141697+0000", 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9699000000000001e-05 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141705+0000", 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7139999999999999e-06 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.373 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142774+0000", 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00106846 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142783+0000", 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6050000000000004e-06 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.147871+0000", 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0050880200000000004 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.147918+0000", 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6817000000000003e-05 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds2 37/31 ECSubWrite(tid=163, reqid=osd.7.0:163, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.108559+0000", 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.184423421, 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0064639540000000001, 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.375 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108559+0000", 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108559+0000", 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108559+0000", 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8099999999999999e-07 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108585+0000", 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6074000000000001e-05 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108646+0000", 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1333000000000004e-05 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108655+0000", 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1689999999999998e-06 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108710+0000", 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4478999999999998e-05 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108718+0000", 2024-09-16T04:08:27.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6359999999999994e-06 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114968+0000", 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0062499280000000001 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115023+0000", 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5154000000000001e-05 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds2 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.109506+0000", 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.183476095, 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.007408492, 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109506+0000", 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109506+0000", 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109507+0000", 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0200000000000001e-07 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109527+0000", 2024-09-16T04:08:27.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0098999999999998e-05 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109573+0000", 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5812000000000001e-05 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109580+0000", 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109636+0000", 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5810000000000003e-05 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109644+0000", 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7540000000000006e-06 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116885+0000", 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0072415550000000002 2024-09-16T04:08:27.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116915+0000", 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9181999999999999e-05 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464424+0000", 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.8285585439999998, 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075752069999999996, 2024-09-16T04:08:27.384 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464424+0000", 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464424+0000", 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464424+0000", 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6899999999999997e-07 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464435+0000", 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0293e-05 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464461+0000", 2024-09-16T04:08:27.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6412000000000001e-05 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464468+0000", 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1300000000000003e-06 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464520+0000", 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1576000000000002e-05 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464531+0000", 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0847e-05 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.471960+0000", 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074292359999999997 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.471999+0000", 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9044000000000001e-05 2024-09-16T04:08:27.388 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.389 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.389 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.389 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 51/48 ECSubWriteReply(tid=36, last_complete=51'22, committed=1, applied=1))", 2024-09-16T04:08:27.389 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:26.486527+0000", 2024-09-16T04:08:27.389 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.80645514200000001, 2024-09-16T04:08:27.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.69968956299999996, 2024-09-16T04:08:27.389 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486527+0000", 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486527+0000", 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486528+0000", 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5899999999999995e-07 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486532+0000", 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4449999999999998e-06 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.185640+0000", 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.69910766300000005 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.185651+0000", 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1423999999999999e-05 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.185718+0000", 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6724999999999996e-05 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.186217+0000", 2024-09-16T04:08:27.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049854700000000001 2024-09-16T04:08:27.393 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.393 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.393 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.393 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.393 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.393 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:27.395 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:27.395 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:27.392+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:27.477 INFO:tasks.rados.rados.0.smithi016.stdout:163: done (4 left) 2024-09-16T04:08:27.477 INFO:tasks.rados.rados.0.smithi016.stdout:164: done (3 left) 2024-09-16T04:08:27.477 INFO:tasks.rados.rados.0.smithi016.stdout:165: done (2 left) 2024-09-16T04:08:27.477 INFO:tasks.rados.rados.0.smithi016.stdout:166: done (1 left) 2024-09-16T04:08:27.477 INFO:tasks.rados.rados.0.smithi016.stdout:168: done (0 left) 2024-09-16T04:08:27.477 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 42 current snap is 12 2024-09-16T04:08:27.477 INFO:tasks.rados.rados.0.smithi016.stdout:169: seq_num 91 ranges {704512=770048,1474560=475136,1949696=294912} 2024-09-16T04:08:27.478 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-16T04:08:27.485 INFO:tasks.rados.rados.0.smithi016.stdout:169: writing smithi01639990-42 from 704512 to 1474560 tid 1 2024-09-16T04:08:27.487 INFO:tasks.rados.rados.0.smithi016.stdout:169: writing smithi01639990-42 from 1474560 to 1949696 tid 2 2024-09-16T04:08:27.490 INFO:tasks.rados.rados.0.smithi016.stdout:169: writing smithi01639990-42 from 1949696 to 2244608 tid 3 2024-09-16T04:08:27.490 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 3 current snap is 12 2024-09-16T04:08:27.490 INFO:tasks.rados.rados.0.smithi016.stdout:170: seq_num 92 ranges {1507328=786432,2293760=491520,2785280=278528} 2024-09-16T04:08:27.495 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:27.492+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:27.502 INFO:tasks.rados.rados.0.smithi016.stdout:170: writing smithi01639990-3 from 1507328 to 2293760 tid 1 2024-09-16T04:08:27.505 INFO:tasks.rados.rados.0.smithi016.stdout:170: writing smithi01639990-3 from 2293760 to 2785280 tid 2 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:170: writing smithi01639990-3 from 2785280 to 3063808 tid 3 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:171: read oid 2 snap 11 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:171: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:172: rollback oid 16 current snap is 12 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 16 to 10 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:169: finishing write tid 1 to smithi01639990-42 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:169: finishing write tid 2 to smithi01639990-42 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:173: read oid 18 snap -1 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:173: expect deleted 2024-09-16T04:08:27.509 INFO:tasks.rados.rados.0.smithi016.stdout:174: setattr oid 44 current snap is 12 2024-09-16T04:08:27.510 INFO:tasks.rados.rados.0.smithi016.stdout:173: done (5 left) 2024-09-16T04:08:27.510 INFO:tasks.rados.rados.0.smithi016.stdout:175: delete oid 28 current snap is 12 2024-09-16T04:08:27.511 INFO:tasks.rados.rados.0.smithi016.stdout:170: finishing write tid 1 to smithi01639990-3 2024-09-16T04:08:27.514 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 44 v 36 (ObjNum 33 snap 0 seq_num 2172044896) dirty exists 2024-09-16T04:08:27.514 INFO:tasks.rados.rados.0.smithi016.stdout:170: finishing write tid 2 to smithi01639990-3 2024-09-16T04:08:27.515 INFO:tasks.rados.rados.0.smithi016.stdout:170: finishing write tid 3 to smithi01639990-3 2024-09-16T04:08:27.515 INFO:tasks.rados.rados.0.smithi016.stdout:170: finishing write tid 4 to smithi01639990-3 2024-09-16T04:08:27.516 INFO:tasks.rados.rados.0.smithi016.stdout:172: finishing rollback tid 0 to smithi01639990-16 2024-09-16T04:08:27.516 INFO:tasks.rados.rados.0.smithi016.stdout:172: finishing rollback tid 1 to smithi01639990-16 2024-09-16T04:08:27.518 INFO:tasks.rados.rados.0.smithi016.stdout:171: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-16T04:08:27.519 INFO:tasks.rados.rados.0.smithi016.stdout:170: finishing write tid 5 to smithi01639990-3 2024-09-16T04:08:27.519 INFO:tasks.rados.rados.0.smithi016.stdout:170: oid 3 updating version 0 to 45 2024-09-16T04:08:27.519 INFO:tasks.rados.rados.0.smithi016.stdout:170: oid 3 updating version 45 to 46 2024-09-16T04:08:27.519 INFO:tasks.rados.rados.0.smithi016.stdout:170: oid 3 updating version 46 to 47 2024-09-16T04:08:27.519 INFO:tasks.rados.rados.0.smithi016.stdout:170: oid 3 updating version 47 to 48 2024-09-16T04:08:27.519 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 3 v 48 (ObjNum 92 snap 12 seq_num 92) dirty exists 2024-09-16T04:08:27.519 INFO:tasks.rados.rados.0.smithi016.stdout:170: left oid 3 (ObjNum 92 snap 12 seq_num 92) 2024-09-16T04:08:27.525 INFO:tasks.rados.rados.0.smithi016.stdout:172: finishing rollback tid 2 to smithi01639990-16 2024-09-16T04:08:27.525 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 16 v 52 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T04:08:27.548 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:27.548 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:27.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.548 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:416 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~294912 in=294912b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:27.548 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:08:27.548 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.056240811000000002, 2024-09-16T04:08:27.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.056261220000000001, 2024-09-16T04:08:27.548 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 416 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:08:27.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490624+0000", 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3069999999999999e-06 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490921+0000", 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00029678400000000002 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490932+0000", 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1239e-05 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490938+0000", 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0379999999999996e-06 2024-09-16T04:08:27.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490980+0000", 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2401999999999998e-05 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491037+0000", 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6768000000000003e-05 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491591+0000", 2024-09-16T04:08:27.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055345000000000004 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492749+0000", 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0011580970000000001 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:417 3.cs0 3:38c68ed3:::smithi01639990-42:head [setxattr _header (57) in=64b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:08:27.553 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.055910792000000001, 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.055979424, 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 417 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:08:27.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2699999999999999e-07 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490956+0000", 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0130000000000002e-06 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490961+0000", 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1510000000000003e-06 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490966+0000", 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7960000000000003e-06 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491934+0000", 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00096816399999999998 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492020+0000", 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5827000000000003e-05 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492479+0000", 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045912299999999997 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492973+0000", 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049418400000000005 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.558 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:418 3.cs0 3:38c68ed3:::smithi01639990-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.055889612999999998, 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.056000055, 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "waiting for rw locks", 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 418 2024-09-16T04:08:27.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8000000000000001e-07 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490976+0000", 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.973e-06 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490983+0000", 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4699999999999999e-06 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490988+0000", 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5509999999999999e-06 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493081+0000", 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020931029999999998 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493165+0000", 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3988999999999998e-05 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.562 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.563 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.563 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.563 INFO:teuthology.orchestra.run.smithi174.stdout: ], 2024-09-16T04:08:27.563 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 3 2024-09-16T04:08:27.563 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:27.564 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:27.596 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:27.594+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:27.655 INFO:tasks.rados.rados.0.smithi016.stdout:169: finishing write tid 3 to smithi01639990-42 2024-09-16T04:08:27.667 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:27.667 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:27.667 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:27.667 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:27.667 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:27.676 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:27.719 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:27.717+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:27.780 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:27.780 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:27.780 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:27.780 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:185 3.5s0 3:a7f99973:::smithi01639990-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.185094+0000", 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.592435748, 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077979190000000004, 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 185 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.781 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185094+0000", 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185094+0000", 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185095+0000", 2024-09-16T04:08:27.782 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7299999999999996e-07 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185098+0000", 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4679999999999999e-06 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185104+0000", 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5189999999999998e-06 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.185109+0000", 2024-09-16T04:08:27.783 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.057e-06 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.188386+0000", 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032768250000000001 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.188443+0000", 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7426000000000002e-05 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.784 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.191927+0000", 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034834190000000002 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.192006+0000", 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9634000000000004e-05 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.192892+0000", 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00088599800000000004 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.785 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 33/27 ECSubWrite(tid=37, reqid=osd.8.0:37, at_version=33'25, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.678823514000001, 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088041119999999994, 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:27.786 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7000000000000001e-07 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098728+0000", 2024-09-16T04:08:27.787 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1432000000000001e-05 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098770+0000", 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1486999999999998e-05 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098780+0000", 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7280000000000007e-06 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.788 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098852+0000", 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2461999999999998e-05 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098862+0000", 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9369999999999999e-06 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107472+0000", 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086102990000000001 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.789 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107511+0000", 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8497000000000001e-05 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.677652901999998, 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011088002, 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.790 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.791 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099878+0000", 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8400000000000003e-07 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099902+0000", 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3629999999999999e-05 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099934+0000", 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2388999999999998e-05 2024-09-16T04:08:27.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099943+0000", 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4800000000000001e-06 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.100212+0000", 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026930999999999998 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.100220+0000", 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8339999999999999e-06 2024-09-16T04:08:27.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110928+0000", 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010708333 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110965+0000", 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7141999999999998e-05 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s7 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.675440582, 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088092050000000005, 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:27.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:27.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.18e-07 2024-09-16T04:08:27.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102106+0000", 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6116999999999998e-05 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102134+0000", 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7963e-05 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102140+0000", 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102224+0000", 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3634000000000006e-05 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102233+0000", 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1609999999999992e-06 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110868+0000", 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086352040000000005 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110899+0000", 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0852999999999998e-05 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:27.799 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.674514747, 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013925085, 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:27.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103016+0000", 2024-09-16T04:08:27.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5499999999999999e-07 2024-09-16T04:08:27.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103038+0000", 2024-09-16T04:08:27.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2416e-05 2024-09-16T04:08:27.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103157+0000", 2024-09-16T04:08:27.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000118439 2024-09-16T04:08:27.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103166+0000", 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9819999999999997e-06 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103225+0000", 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8996000000000002e-05 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103235+0000", 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9909999999999997e-06 2024-09-16T04:08:27.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116899+0000", 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013664549999999999 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116941+0000", 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1455999999999997e-05 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.804 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.674489445999999, 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013929731000000001, 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5899999999999998e-07 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103057+0000", 2024-09-16T04:08:27.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5778e-05 2024-09-16T04:08:27.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103193+0000", 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0001359 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103199+0000", 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1279999999999996e-06 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107717+0000", 2024-09-16T04:08:27.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045183769999999996 2024-09-16T04:08:27.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107726+0000", 2024-09-16T04:08:27.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5420000000000005e-06 2024-09-16T04:08:27.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116950+0000", 2024-09-16T04:08:27.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092240229999999996 2024-09-16T04:08:27.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116970+0000", 2024-09-16T04:08:27.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0724e-05 2024-09-16T04:08:27.811 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.811 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.811 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.811 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s5 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:27.811 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:27.811 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.673740643999999, 2024-09-16T04:08:27.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070187460000000002, 2024-09-16T04:08:27.812 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.812 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.812 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.812 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:27.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:27.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.814 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.814 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103790+0000", 2024-09-16T04:08:27.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5000000000000001e-07 2024-09-16T04:08:27.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.814 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.814 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103801+0000", 2024-09-16T04:08:27.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0927999999999999e-05 2024-09-16T04:08:27.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103842+0000", 2024-09-16T04:08:27.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0649999999999999e-05 2024-09-16T04:08:27.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103851+0000", 2024-09-16T04:08:27.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8920000000000006e-06 2024-09-16T04:08:27.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.816 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.816 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103944+0000", 2024-09-16T04:08:27.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2705000000000001e-05 2024-09-16T04:08:27.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.817 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.817 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103955+0000", 2024-09-16T04:08:27.817 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1282e-05 2024-09-16T04:08:27.817 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.817 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.817 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.817 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110771+0000", 2024-09-16T04:08:27.817 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0068159270000000003 2024-09-16T04:08:27.817 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.818 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.818 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110808+0000", 2024-09-16T04:08:27.818 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7512000000000003e-05 2024-09-16T04:08:27.818 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.818 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.818 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.818 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.819 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s5 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:27.819 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:27.819 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.670625756, 2024-09-16T04:08:27.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0080941920000000001, 2024-09-16T04:08:27.819 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.819 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.819 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.819 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:27.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.820 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:27.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106905+0000", 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4999999999999999e-07 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106926+0000", 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1202000000000001e-05 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107015+0000", 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9407999999999995e-05 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107034+0000", 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8896999999999999e-05 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107499+0000", 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046474800000000002 2024-09-16T04:08:27.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107509+0000", 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0390999999999999e-05 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114906+0000", 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073969409999999998 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114999+0000", 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2454999999999995e-05 2024-09-16T04:08:27.823 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds7 37/31 ECSubWrite(tid=163, reqid=osd.7.0:163, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.668972553, 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084527260000000007, 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:27.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0100000000000004e-07 2024-09-16T04:08:27.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108578+0000", 2024-09-16T04:08:27.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9366999999999999e-05 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108616+0000", 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8056999999999998e-05 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108624+0000", 2024-09-16T04:08:27.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.824e-06 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108684+0000", 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0019999999999998e-05 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108693+0000", 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2599999999999994e-06 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116974+0000", 2024-09-16T04:08:27.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082815990000000006 2024-09-16T04:08:27.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.117010+0000", 2024-09-16T04:08:27.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5898000000000001e-05 2024-09-16T04:08:27.829 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds7 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.667998648999999, 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091377019999999993, 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6499999999999998e-07 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.831 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109554+0000", 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2030000000000001e-05 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109587+0000", 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3019999999999999e-05 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109603+0000", 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5678000000000001e-05 2024-09-16T04:08:27.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110514+0000", 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00091074700000000005 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110523+0000", 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4639999999999995e-06 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118636+0000", 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008112157 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118669+0000", 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3741000000000001e-05 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:27.834 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.666115622, 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0072197169999999996, 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111415+0000", 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7400000000000003e-07 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111431+0000", 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5305e-05 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111457+0000", 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6067e-05 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111462+0000", 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3480000000000003e-06 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111516+0000", 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3897000000000003e-05 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111526+0000", 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7440000000000002e-06 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118592+0000", 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070660710000000002 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118634+0000", 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2611000000000002e-05 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:27.839 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.333058230000001, 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011544518, 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444473+0000", 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4499999999999996e-07 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447203+0000", 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002729834 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447277+0000", 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4078999999999993e-05 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447283+0000", 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4169999999999997e-06 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447458+0000", 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00017450300000000001 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447469+0000", 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1062e-05 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.455978+0000", 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085090730000000007 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456017+0000", 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8804999999999999e-05 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.843 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:351 3.cs0 3:338c6e39:::smithi01639990-33:6 [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.446452+0000", 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.331078308, 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0074593050000000003, 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:27.844 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 351 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446452+0000", 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446452+0000", 2024-09-16T04:08:27.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446452+0000", 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2300000000000001e-07 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446456+0000", 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.242e-06 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446462+0000", 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6329999999999999e-06 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446475+0000", 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2439999999999999e-05 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446992+0000", 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00051762900000000003 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447118+0000", 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012605900000000001 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453911+0000", 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067927789999999997 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:352 3.cs0 3:338c6e39:::smithi01639990-33:6 [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:27.848 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.331046371999999, 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078540529999999997, 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 352 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:27.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:27.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9200000000000001e-07 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446487+0000", 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0489999999999999e-06 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446494+0000", 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5180000000000001e-06 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446499+0000", 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9010000000000002e-06 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447665+0000", 2024-09-16T04:08:27.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0011659120000000001 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447747+0000", 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2785000000000001e-05 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.454338+0000", 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065903960000000001 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.854 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 42/27 ECSubWrite(tid=51, reqid=osd.11.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=42'32))", 2024-09-16T04:08:27.854 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.448304+0000", 2024-09-16T04:08:27.854 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.329226413000001, 2024-09-16T04:08:27.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077603150000000003, 2024-09-16T04:08:27.854 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.854 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448304+0000", 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448304+0000", 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448305+0000", 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4300000000000002e-07 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448319+0000", 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4438e-05 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448347+0000", 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8070999999999999e-05 2024-09-16T04:08:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448357+0000", 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7960000000000007e-06 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448412+0000", 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5300000000000002e-05 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448506+0000", 2024-09-16T04:08:27.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3948000000000002e-05 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456027+0000", 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075212589999999998 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456064+0000", 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.676e-05 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.858 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/27 ECSubWriteReply(tid=88, last_complete=42'39, committed=1, applied=1))", 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.329084213, 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49630058100000002, 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.860 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448447+0000", 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448451+0000", 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3030000000000001e-06 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944646+0000", 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49619510999999999 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944655+0000", 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8480000000000007e-06 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944717+0000", 2024-09-16T04:08:27.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2065999999999998e-05 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944747+0000", 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9505999999999998e-05 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.450032+0000", 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.327497738, 2024-09-16T04:08:27.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088454970000000008, 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450032+0000", 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450032+0000", 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450033+0000", 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2399999999999998e-07 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451065+0000", 2024-09-16T04:08:27.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010323089999999999 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451134+0000", 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8096999999999995e-05 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451140+0000", 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5939999999999999e-06 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451197+0000", 2024-09-16T04:08:27.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6665e-05 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451206+0000", 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3530000000000001e-06 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.458843+0000", 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0076371549999999996 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.458878+0000", 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4700000000000003e-05 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDScrubReserve(3.0s7,QREQUEST,e:47,from:11(0),reservation_nonce:4)", 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.7503623990000001, 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17961150300000001, 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027169+0000", 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:08:27.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027173+0000", 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1300000000000003e-06 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206636+0000", 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17946284700000001 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206644+0000", 2024-09-16T04:08:27.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9650000000000008e-06 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206712+0000", 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8215999999999999e-05 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206735+0000", 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2789000000000002e-05 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206779+0000", 2024-09-16T04:08:27.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4669000000000002e-05 2024-09-16T04:08:27.872 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.872 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.872 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.872 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:414 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~770048] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:27.872 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.29298810199999997, 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012099706, 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 414 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:08:27.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484544+0000", 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485808+0000", 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012637099999999999 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485825+0000", 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7722000000000001e-05 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485834+0000", 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9749999999999996e-06 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485905+0000", 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1131999999999996e-05 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.486012+0000", 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000106214 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487551+0000", 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015389830000000001 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.489803+0000", 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0022524839999999999 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496554+0000", 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067513879999999997 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496642+0000", 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7385000000000004e-05 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:415 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~475136] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.487079+0000", 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.29045162699999999, 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097770519999999996, 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:27.878 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 415 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487079+0000", 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487079+0000", 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487079+0000", 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6199999999999998e-07 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487619+0000", 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053991599999999999 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487633+0000", 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4158000000000001e-05 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487654+0000", 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0669000000000001e-05 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.488619+0000", 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00096477599999999998 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.488718+0000", 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9809000000000006e-05 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.489445+0000", 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072619099999999995 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490031+0000", 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000586514 2024-09-16T04:08:27.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496790+0000", 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.006758633 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496856+0000", 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5823999999999999e-05 2024-09-16T04:08:27.883 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.884 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.884 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.884 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:27.884 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:27.884 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:27.885 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:27.886 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:27.883+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:27.986 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:27.984+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:28.030 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:28.030 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:28.030 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:28.030 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:28.039 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:28.124 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:28.121+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:28.165 INFO:tasks.rados.rados.0.smithi016.stdout:169: finishing write tid 4 to smithi01639990-42 2024-09-16T04:08:28.166 INFO:tasks.rados.rados.0.smithi016.stdout:169: finishing write tid 5 to smithi01639990-42 2024-09-16T04:08:28.166 INFO:tasks.rados.rados.0.smithi016.stdout:169: oid 42 updating version 0 to 27 2024-09-16T04:08:28.166 INFO:tasks.rados.rados.0.smithi016.stdout:169: oid 42 version 27 is already newer than 26 2024-09-16T04:08:28.166 INFO:tasks.rados.rados.0.smithi016.stdout:169: oid 42 version 27 is already newer than 25 2024-09-16T04:08:28.166 INFO:tasks.rados.rados.0.smithi016.stdout:169: oid 42 version 27 is already newer than 24 2024-09-16T04:08:28.166 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 42 v 27 (ObjNum 91 snap 12 seq_num 91) dirty exists 2024-09-16T04:08:28.166 INFO:tasks.rados.rados.0.smithi016.stdout:169: left oid 42 (ObjNum 91 snap 12 seq_num 91) 2024-09-16T04:08:28.183 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:28.183 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:28.183 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:28.183 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:28.183 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:28.188 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:28.187+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:28.191 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:28.289 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:28.287+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:193 3.as0 3:5d02464a:::smithi01639990-40:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.137678610999998, 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041575429999999997, 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.337 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 193 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196587+0000", 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196589+0000", 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8160000000000001e-06 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196600+0000", 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1545e-05 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196613+0000", 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.34e-05 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196621+0000", 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4909999999999999e-06 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196682+0000", 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1074000000000005e-05 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.196776+0000", 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4102000000000001e-05 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.197563+0000", 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000786446 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.341 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.198374+0000", 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081128199999999998 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200668+0000", 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0022936979999999998 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.200744+0000", 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6688999999999997e-05 2024-09-16T04:08:28.342 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.230561932000001, 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0068480019999999997, 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.343 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.344 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103704+0000", 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7100000000000001e-07 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103727+0000", 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3694999999999999e-05 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103810+0000", 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3091999999999997e-05 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.345 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103816+0000", 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4519999999999996e-06 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103971+0000", 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000154993 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103982+0000", 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0933e-05 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.346 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110491+0000", 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065096800000000003 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110551+0000", 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9985999999999998e-05 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.347 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.230512594, 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067325809999999996, 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.348 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103754+0000", 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5899999999999995e-07 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103768+0000", 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4562999999999999e-05 2024-09-16T04:08:28.349 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103805+0000", 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6730000000000002e-05 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103813+0000", 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.350 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103868+0000", 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4475000000000003e-05 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103877+0000", 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7099999999999996e-06 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110444+0000", 2024-09-16T04:08:28.351 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065675170000000001 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110485+0000", 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1177000000000002e-05 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/27 ECSubWrite(tid=84, reqid=client.4440.0:215, at_version=33'14, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:28.352 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.195552974999998, 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042300860000000001, 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:28.353 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.138712+0000", 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5900000000000001e-07 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139362+0000", 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064961800000000003 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.354 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139387+0000", 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5369999999999999e-05 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139394+0000", 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2190000000000001e-06 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.355 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139458+0000", 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4603000000000005e-05 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.139470+0000", 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1810999999999999e-05 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142910+0000", 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003439896 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.356 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.142942+0000", 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2410000000000003e-05 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 33/27 ECSubWrite(tid=85, reqid=client.4440.0:216, at_version=33'15, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.193350702, 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043866510000000001, 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.357 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.358 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.140915+0000", 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141225+0000", 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00030972200000000001 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141258+0000", 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3228000000000001e-05 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.359 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141267+0000", 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5590000000000006e-06 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141327+0000", 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0100999999999999e-05 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.360 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.141338+0000", 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0745e-05 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145260+0000", 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039227610000000003 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.145301+0000", 2024-09-16T04:08:28.361 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0884999999999999e-05 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 37/27 ECSubWrite(tid=71, reqid=osd.5.0:71, at_version=37'24, trim_to=0'0, roll_forward_to=35'22))", 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.231574782999999, 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040595509999999998, 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.362 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102691+0000", 2024-09-16T04:08:28.363 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4000000000000001e-07 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102711+0000", 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0409999999999999e-05 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103362+0000", 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065125100000000004 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.364 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103372+0000", 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7340000000000003e-06 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103464+0000", 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1663999999999995e-05 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103474+0000", 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8630000000000003e-06 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.365 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106711+0000", 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0032372669999999998 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106750+0000", 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9221999999999998e-05 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.366 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=70, reqid=osd.5.0:70, at_version=37'20, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.231552662, 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0053540310000000004, 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.367 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102713+0000", 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3799999999999999e-07 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.102721+0000", 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6500000000000002e-06 2024-09-16T04:08:28.368 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103774+0000", 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010529070000000001 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103806+0000", 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1303999999999997e-05 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.369 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103868+0000", 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1937e-05 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103877+0000", 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6199999999999994e-06 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108037+0000", 2024-09-16T04:08:28.370 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041593389999999997 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108067+0000", 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0136000000000001e-05 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/27 ECSubWrite(tid=124, reqid=osd.3.0:124, at_version=37'22, trim_to=0'0, roll_forward_to=35'20))", 2024-09-16T04:08:28.371 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.230928294, 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070925160000000001, 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.372 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6999999999999999e-07 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103359+0000", 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1835999999999999e-05 2024-09-16T04:08:28.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103683+0000", 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00032427399999999999 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103693+0000", 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7380000000000006e-06 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103756+0000", 2024-09-16T04:08:28.374 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3115999999999997e-05 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103766+0000", 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0043e-05 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110407+0000", 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0066407230000000003 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.375 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110430+0000", 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2616000000000002e-05 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.230098021, 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008202661, 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.376 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104168+0000", 2024-09-16T04:08:28.377 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0200000000000001e-07 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104188+0000", 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9933999999999999e-05 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104232+0000", 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4196000000000002e-05 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104241+0000", 2024-09-16T04:08:28.378 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7979999999999995e-06 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104928+0000", 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687124 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104937+0000", 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5269999999999998e-06 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.379 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112328+0000", 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073914510000000003 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112370+0000", 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1928999999999998e-05 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.bs7 37/27 ECSubWrite(tid=127, reqid=osd.3.0:127, at_version=37'22, trim_to=0'0, roll_forward_to=35'17))", 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.105041+0000", 2024-09-16T04:08:28.380 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.229224133000001, 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046635560000000001, 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105041+0000", 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.381 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105041+0000", 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105042+0000", 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9299999999999997e-07 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105073+0000", 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1041999999999999e-05 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.382 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105110+0000", 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7113000000000002e-05 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105118+0000", 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4330000000000004e-06 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105253+0000", 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000134191 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.383 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.105264+0000", 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1827e-05 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109656+0000", 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043918459999999996 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109705+0000", 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8411000000000001e-05 2024-09-16T04:08:28.384 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.227378413, 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082659529999999995, 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.385 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1600000000000002e-07 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.386 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106910+0000", 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2954000000000001e-05 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106990+0000", 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9788000000000002e-05 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106997+0000", 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4379999999999998e-06 2024-09-16T04:08:28.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107267+0000", 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026999599999999999 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107276+0000", 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9530000000000005e-06 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115126+0000", 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078491940000000003 2024-09-16T04:08:28.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115153+0000", 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7214000000000001e-05 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 37/27 ECSubWrite(tid=73, reqid=osd.5.0:71, at_version=0'0, trim_to=0'0, roll_forward_to=37'24))", 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.110673+0000", 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.223592261, 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044773169999999998, 2024-09-16T04:08:28.389 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110673+0000", 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110673+0000", 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.390 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110674+0000", 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6199999999999997e-07 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110685+0000", 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1221e-05 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110720+0000", 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5049000000000003e-05 2024-09-16T04:08:28.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110729+0000", 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3370000000000006e-06 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110800+0000", 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0425000000000004e-05 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110810+0000", 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.009e-05 2024-09-16T04:08:28.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115120+0000", 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043095920000000001 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115150+0000", 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0840999999999998e-05 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.4s6 37/27 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=37'15, trim_to=0'0, roll_forward_to=35'13))", 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.222892443999999, 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0054036320000000002, 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111373+0000", 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899999999999999e-07 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.395 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111387+0000", 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3501e-05 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111416+0000", 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9377999999999999e-05 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111421+0000", 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2889999999999997e-06 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.396 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111470+0000", 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9162000000000001e-05 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.111479+0000", 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9900000000000003e-06 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116730+0000", 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0052506219999999999 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.397 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116777+0000", 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6551000000000002e-05 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=74, reqid=osd.5.0:72, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.221465634999999, 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040065320000000001, 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.398 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112800+0000", 2024-09-16T04:08:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5400000000000001e-07 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112809+0000", 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9050000000000003e-06 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112841+0000", 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1983999999999997e-05 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112849+0000", 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3750000000000003e-06 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112915+0000", 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5599000000000007e-05 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112922+0000", 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.401 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116782+0000", 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038605890000000002 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116806+0000", 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3760999999999998e-05 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:28.402 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.884325188, 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091491550000000008, 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:28.403 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5800000000000001e-07 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450512+0000", 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057160000000000002 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.404 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450541+0000", 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8835000000000001e-05 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450549+0000", 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3080000000000008e-06 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450607+0000", 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7763999999999998e-05 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.405 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450616+0000", 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459045+0000", 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084290709999999998 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459089+0000", 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4233000000000002e-05 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.1s6 42/27 ECSubWrite(tid=41, reqid=osd.0.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=42'11))", 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.451624+0000", 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.882640990000001, 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044678180000000001, 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451624+0000", 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451624+0000", 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451625+0000", 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7100000000000001e-07 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451635+0000", 2024-09-16T04:08:28.408 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.825e-06 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451672+0000", 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7134000000000001e-05 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451680+0000", 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.371e-06 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451737+0000", 2024-09-16T04:08:28.409 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6675000000000002e-05 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451744+0000", 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.041e-06 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456050+0000", 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043059860000000004 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.410 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456092+0000", 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2014999999999997e-05 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=160, reqid=client.4440.0:347, at_version=42'32, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.881703106, 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065520760000000004, 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.411 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452563+0000", 2024-09-16T04:08:28.412 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6300000000000005e-07 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452621+0000", 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8488e-05 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452654+0000", 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2750999999999999e-05 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.413 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452659+0000", 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.293e-06 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452730+0000", 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0173999999999996e-05 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452738+0000", 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6260000000000001e-06 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.414 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459095+0000", 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0063566530000000003 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459114+0000", 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9428000000000001e-05 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=161, reqid=client.4440.0:348, at_version=42'33, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:28.415 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.8816405, 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065540390000000002, 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:28.416 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6300000000000001e-07 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452632+0000", 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6699999999999997e-06 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.417 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452678+0000", 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5760000000000002e-05 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452682+0000", 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.036e-06 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453280+0000", 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059861699999999996 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.418 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453289+0000", 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9590000000000001e-06 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459117+0000", 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058281770000000004 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459179+0000", 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1556999999999994e-05 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=167, reqid=client.4440.0:355, at_version=42'37, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.588912+0000", 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.745352945, 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004899128, 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.588912+0000", 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.588912+0000", 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.588913+0000", 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4199999999999995e-07 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589099+0000", 2024-09-16T04:08:28.421 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00018556200000000001 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589134+0000", 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5553999999999999e-05 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589142+0000", 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0979999999999994e-06 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.422 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589199+0000", 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6586999999999999e-05 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.589210+0000", 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.078e-05 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.593773+0000", 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045635179999999999 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.423 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.593811+0000", 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8287000000000001e-05 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:362 3.as0 3:5e52341f:::smithi01639990-38:head [] snapc 8=[6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:18.439764+0000", 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.8945012479999992, 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004077855, 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.424 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 362 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439764+0000", 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439764+0000", 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439765+0000", 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2839999999999999e-06 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439771+0000", 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0580000000000003e-06 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439784+0000", 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2301e-05 2024-09-16T04:08:28.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439790+0000", 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1020000000000002e-06 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439842+0000", 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2308999999999997e-05 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.439848+0000", 2024-09-16T04:08:28.427 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0469999999999999e-06 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.443795+0000", 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039468209999999997 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:18.443842+0000", 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6932999999999999e-05 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.428 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.429 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.429 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:28.430 INFO:tasks.rados.rados.0.smithi016.stdout:169: done (5 left) 2024-09-16T04:08:28.430 INFO:tasks.rados.rados.0.smithi016.stdout:170: done (4 left) 2024-09-16T04:08:28.430 INFO:tasks.rados.rados.0.smithi016.stdout:171: done (3 left) 2024-09-16T04:08:28.430 INFO:tasks.rados.rados.0.smithi016.stdout:172: done (2 left) 2024-09-16T04:08:28.430 INFO:tasks.rados.rados.0.smithi016.stdout:174: done (1 left) 2024-09-16T04:08:28.430 INFO:tasks.rados.rados.0.smithi016.stdout:175: done (0 left) 2024-09-16T04:08:28.431 INFO:tasks.rados.rados.0.smithi016.stdout:176: rmattr oid 3 current snap is 12 2024-09-16T04:08:28.431 INFO:tasks.rados.rados.0.smithi016.stdout:176: done (0 left) 2024-09-16T04:08:28.431 INFO:tasks.rados.rados.0.smithi016.stdout:177: rmattr oid 22 current snap is 12 2024-09-16T04:08:28.431 INFO:tasks.rados.rados.0.smithi016.stdout:178: snap_remove snap 10 2024-09-16T04:08:28.431 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:28.432 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:28.429+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:28.482 INFO:tasks.rados.rados.0.smithi016.stdout:178: done (1 left) 2024-09-16T04:08:28.482 INFO:tasks.rados.rados.0.smithi016.stdout:179: read oid 50 snap 9 2024-09-16T04:08:28.482 INFO:tasks.rados.rados.0.smithi016.stdout:179: expect deleted 2024-09-16T04:08:28.482 INFO:tasks.rados.rados.0.smithi016.stdout:180: rollback oid 40 current snap is 12 2024-09-16T04:08:28.482 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 40 to 9 2024-09-16T04:08:28.482 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T04:08:28.482 INFO:tasks.rados.rados.0.smithi016.stdout:177: done (2 left) 2024-09-16T04:08:28.483 INFO:tasks.rados.rados.0.smithi016.stdout:181: copy_from oid 48 from oid 18 current snap is 12 2024-09-16T04:08:28.483 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 47 current snap is 12 2024-09-16T04:08:28.483 INFO:tasks.rados.rados.0.smithi016.stdout:182: seq_num 94 ranges {1802240=425984,2228224=786432,3014656=376832} 2024-09-16T04:08:28.495 INFO:tasks.rados.rados.0.smithi016.stdout:182: writing smithi01639990-47 from 1802240 to 2228224 tid 1 2024-09-16T04:08:28.500 INFO:tasks.rados.rados.0.smithi016.stdout:182: writing smithi01639990-47 from 2228224 to 3014656 tid 2 2024-09-16T04:08:28.503 INFO:tasks.rados.rados.0.smithi016.stdout:182: writing smithi01639990-47 from 3014656 to 3391488 tid 3 2024-09-16T04:08:28.503 INFO:tasks.rados.rados.0.smithi016.stdout:183: rmattr oid 41 current snap is 12 2024-09-16T04:08:28.503 INFO:tasks.rados.rados.0.smithi016.stdout:180: finishing rollback tid 1 to smithi01639990-40 2024-09-16T04:08:28.504 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 40 v 0 (ObjNum 129 snap 0 seq_num 2170701304) dirty dne 2024-09-16T04:08:28.504 INFO:tasks.rados.rados.0.smithi016.stdout:181: finishing copy_from to smithi01639990-48 2024-09-16T04:08:28.504 INFO:tasks.rados.rados.0.smithi016.stdout:181: got expected ENOENT (src dne) 2024-09-16T04:08:28.504 INFO:tasks.rados.rados.0.smithi016.stdout:179: done (4 left) 2024-09-16T04:08:28.504 INFO:tasks.rados.rados.0.smithi016.stdout:180: done (3 left) 2024-09-16T04:08:28.504 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 37 current snap is 12 2024-09-16T04:08:28.504 INFO:tasks.rados.rados.0.smithi016.stdout:181: finishing copy_from racing read to smithi01639990-48 2024-09-16T04:08:28.504 INFO:tasks.rados.rados.0.smithi016.stdout:184: seq_num 95 ranges {606208=442368,1048576=540672,1589248=622592} 2024-09-16T04:08:28.509 INFO:tasks.rados.rados.0.smithi016.stdout:184: writing smithi01639990-37 from 606208 to 1048576 tid 1 2024-09-16T04:08:28.513 INFO:tasks.rados.rados.0.smithi016.stdout:184: writing smithi01639990-37 from 1048576 to 1589248 tid 2 2024-09-16T04:08:28.517 INFO:tasks.rados.rados.0.smithi016.stdout:184: writing smithi01639990-37 from 1589248 to 2211840 tid 3 2024-09-16T04:08:28.517 INFO:tasks.rados.rados.0.smithi016.stdout:181: done (3 left) 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:185: rmattr oid 16 current snap is 12 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:185: done (3 left) 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:186: rollback oid 23 current snap is 12 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 23 to 9 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:182: finishing write tid 1 to smithi01639990-47 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:182: finishing write tid 2 to smithi01639990-47 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 41 v 56 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:182: finishing write tid 3 to smithi01639990-47 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:183: done (3 left) 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 38 current snap is 12 2024-09-16T04:08:28.518 INFO:tasks.rados.rados.0.smithi016.stdout:187: seq_num 96 ranges {622592=458752,1081344=688128,1769472=475136} 2024-09-16T04:08:28.523 INFO:tasks.rados.rados.0.smithi016.stdout:187: writing smithi01639990-38 from 622592 to 1081344 tid 1 2024-09-16T04:08:28.527 INFO:tasks.rados.rados.0.smithi016.stdout:187: writing smithi01639990-38 from 1081344 to 1769472 tid 2 2024-09-16T04:08:28.531 INFO:tasks.rados.rados.0.smithi016.stdout:187: writing smithi01639990-38 from 1769472 to 2244608 tid 3 2024-09-16T04:08:28.531 INFO:tasks.rados.rados.0.smithi016.stdout:188: read oid 49 snap -1 2024-09-16T04:08:28.531 INFO:tasks.rados.rados.0.smithi016.stdout:188: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-16T04:08:28.531 INFO:tasks.rados.rados.0.smithi016.stdout:189: copy_from oid 1 from oid 34 current snap is 12 2024-09-16T04:08:28.531 INFO:tasks.rados.rados.0.smithi016.stdout:184: finishing write tid 1 to smithi01639990-37 2024-09-16T04:08:28.531 INFO:tasks.rados.rados.0.smithi016.stdout:184: finishing write tid 2 to smithi01639990-37 2024-09-16T04:08:28.531 INFO:tasks.rados.rados.0.smithi016.stdout:184: finishing write tid 3 to smithi01639990-37 2024-09-16T04:08:28.531 INFO:tasks.rados.rados.0.smithi016.stdout:187: finishing write tid 1 to smithi01639990-38 2024-09-16T04:08:28.532 INFO:tasks.rados.rados.0.smithi016.stdout:190: read oid 41 snap -1 2024-09-16T04:08:28.532 INFO:tasks.rados.rados.0.smithi016.stdout:190: expect (ObjNum 61 snap 2 seq_num 61) 2024-09-16T04:08:28.532 INFO:tasks.rados.rados.0.smithi016.stdout:191: delete oid 32 current snap is 12 2024-09-16T04:08:28.532 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:28.527+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:28.533 INFO:tasks.rados.rados.0.smithi016.stdout:187: finishing write tid 2 to smithi01639990-38 2024-09-16T04:08:28.535 INFO:tasks.rados.rados.0.smithi016.stdout:187: finishing write tid 3 to smithi01639990-38 2024-09-16T04:08:28.535 INFO:tasks.rados.rados.0.smithi016.stdout:187: finishing write tid 4 to smithi01639990-38 2024-09-16T04:08:28.537 INFO:tasks.rados.rados.0.smithi016.stdout:187: finishing write tid 5 to smithi01639990-38 2024-09-16T04:08:28.537 INFO:tasks.rados.rados.0.smithi016.stdout:187: oid 38 updating version 0 to 15 2024-09-16T04:08:28.537 INFO:tasks.rados.rados.0.smithi016.stdout:187: oid 38 updating version 15 to 17 2024-09-16T04:08:28.537 INFO:tasks.rados.rados.0.smithi016.stdout:187: oid 38 version 17 is already newer than 16 2024-09-16T04:08:28.537 INFO:tasks.rados.rados.0.smithi016.stdout:187: oid 38 updating version 17 to 18 2024-09-16T04:08:28.537 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 38 v 18 (ObjNum 96 snap 12 seq_num 96) dirty exists 2024-09-16T04:08:28.537 INFO:tasks.rados.rados.0.smithi016.stdout:187: left oid 38 (ObjNum 96 snap 12 seq_num 96) 2024-09-16T04:08:28.538 INFO:tasks.rados.rados.0.smithi016.stdout:187: done (7 left) 2024-09-16T04:08:28.538 INFO:tasks.rados.rados.0.smithi016.stdout:191: done (6 left) 2024-09-16T04:08:28.538 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 38 current snap is 12 2024-09-16T04:08:28.538 INFO:tasks.rados.rados.0.smithi016.stdout:192: seq_num 98 ranges {2244608=491520,2736128=589824,3325952=557056,3883008=16384} 2024-09-16T04:08:28.553 INFO:tasks.rados.rados.0.smithi016.stdout:192: writing smithi01639990-38 from 2244608 to 2736128 tid 1 2024-09-16T04:08:28.556 INFO:tasks.rados.rados.0.smithi016.stdout:192: writing smithi01639990-38 from 2736128 to 3325952 tid 2 2024-09-16T04:08:28.560 INFO:tasks.rados.rados.0.smithi016.stdout:192: writing smithi01639990-38 from 3325952 to 3883008 tid 3 2024-09-16T04:08:28.560 INFO:tasks.rados.rados.0.smithi016.stdout:192: writing smithi01639990-38 from 3883008 to 3899392 tid 4 2024-09-16T04:08:28.560 INFO:tasks.rados.rados.0.smithi016.stdout:193: snap_create 2024-09-16T04:08:28.560 INFO:tasks.rados.rados.0.smithi016.stdout:190: expect (ObjNum 61 snap 2 seq_num 61) 2024-09-16T04:08:28.598 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:28.599 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:28.599 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:28.599 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:28.607 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:28.617 INFO:tasks.rados.rados.0.smithi016.stdout:188: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-16T04:08:28.631 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:28.628+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:28.689 INFO:tasks.rados.rados.0.smithi016.stdout:189: finishing copy_from racing read to smithi01639990-1 2024-09-16T04:08:28.689 INFO:tasks.rados.rados.0.smithi016.stdout:189: finishing copy_from to smithi01639990-1 2024-09-16T04:08:28.689 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 1 v 19 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-09-16T04:08:28.689 INFO:tasks.rados.rados.0.smithi016.stdout:192: finishing write tid 1 to smithi01639990-38 2024-09-16T04:08:28.689 INFO:tasks.rados.rados.0.smithi016.stdout:192: finishing write tid 2 to smithi01639990-38 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: finishing write tid 3 to smithi01639990-38 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: finishing write tid 4 to smithi01639990-38 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: finishing write tid 5 to smithi01639990-38 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: finishing write tid 6 to smithi01639990-38 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: oid 38 updating version 0 to 19 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: oid 38 updating version 19 to 20 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: oid 38 updating version 20 to 21 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: oid 38 updating version 21 to 22 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: oid 38 updating version 22 to 23 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 38 v 23 (ObjNum 98 snap 12 seq_num 98) dirty exists 2024-09-16T04:08:28.690 INFO:tasks.rados.rados.0.smithi016.stdout:192: left oid 38 (ObjNum 98 snap 12 seq_num 98) 2024-09-16T04:08:28.711 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:28.711 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:28.711 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:28.711 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:28.711 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:28.720 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:28.732 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:28.730+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:28.824 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.881329909999998, 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.825 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:28.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:28.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:28.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 37.898925667, 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.834 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:28.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:28.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:28.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:28.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 37.888563161, 2024-09-16T04:08:28.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:28.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:28.844 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 36.391975109000001, 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:28.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:28.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:28.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:28.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:28.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:28.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:28.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:28.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 36.368462207, 2024-09-16T04:08:28.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092583400000000003, 2024-09-16T04:08:28.852 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.852 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.852 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.852 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.852 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:28.852 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:28.852 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 141 2024-09-16T04:08:28.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.852 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:28.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453591+0000", 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.37e-07 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453604+0000", 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2658e-05 2024-09-16T04:08:28.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453611+0000", 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0650000000000001e-06 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.453616+0000", 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9099999999999996e-06 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459323+0000", 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057072879999999996 2024-09-16T04:08:28.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459369+0000", 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6436000000000001e-05 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.459442+0000", 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2459000000000005e-05 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:28.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.460087+0000", 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064517999999999997 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462796+0000", 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027087639999999998 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462827+0000", 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0827000000000001e-05 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.462849+0000", 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2215999999999999e-05 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:161 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.937505774000002, 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088608530000000001, 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.858 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 161 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884547+0000", 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884548+0000", 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5599999999999995e-07 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884603+0000", 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5615999999999997e-05 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884612+0000", 2024-09-16T04:08:28.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6589999999999996e-06 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884619+0000", 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4630000000000004e-06 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889203+0000", 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045841989999999997 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889242+0000", 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8713e-05 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.889304+0000", 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1858000000000003e-05 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892819+0000", 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003515107 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:28.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893354+0000", 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053514000000000003 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893377+0000", 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2265000000000001e-05 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893408+0000", 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1276999999999999e-05 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.863 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.937426524999999, 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094233230000000008, 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 162 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.864 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884627+0000", 2024-09-16T04:08:28.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884629+0000", 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.187e-06 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884632+0000", 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.501e-06 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884635+0000", 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7389999999999999e-06 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892223+0000", 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075881029999999997 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892291+0000", 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7508000000000005e-05 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:28.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892412+0000", 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012124 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893979+0000", 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015676069999999999 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893987+0000", 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.306e-06 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894013+0000", 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6305999999999999e-05 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894050+0000", 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6492999999999998e-05 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:28.869 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.937414154000002, 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095013200000000006, 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 163 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2100000000000001e-07 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884642+0000", 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.478e-06 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884646+0000", 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.292e-06 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884648+0000", 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6230000000000001e-06 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892847+0000", 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0081990580000000004 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892891+0000", 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3628999999999999e-05 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892959+0000", 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8151000000000005e-05 2024-09-16T04:08:28.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894055+0000", 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001095927 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894088+0000", 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3016999999999999e-05 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894117+0000", 2024-09-16T04:08:28.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9204999999999999e-05 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894140+0000", 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2719000000000001e-05 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.937400615999998, 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:28.875 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:28.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:28.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:28.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:28.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.937346554999998, 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.881 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:28.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:28.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:28.885 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.886 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.713078017000001, 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:28.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:28.890 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.135144057000002, 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.894 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s2 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.722369828000001, 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010866819, 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099684+0000", 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.004e-06 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099714+0000", 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9807000000000001e-05 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099769+0000", 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5306000000000002e-05 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099781+0000", 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2091e-05 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099835+0000", 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4058000000000003e-05 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099845+0000", 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5729999999999997e-06 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110509+0000", 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010664339 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.898 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110550+0000", 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640999999999999e-05 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.718330911999999, 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.899 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.902 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.903 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.716943497999999, 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.904 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.905 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.906 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.907 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.714676442999998, 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099172860000000009, 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.908 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107377+0000", 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8900000000000001e-07 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.909 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107437+0000", 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0628000000000003e-05 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107526+0000", 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8873000000000001e-05 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107535+0000", 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0489999999999993e-06 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.910 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111543+0000", 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040080810000000001 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111554+0000", 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0321e-05 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117271+0000", 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057176529999999996 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.911 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117294+0000", 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2391999999999998e-05 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 42/27 ECSubWrite(tid=87, reqid=osd.5.0:87, at_version=42'39, trim_to=0'0, roll_forward_to=41'35))", 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.436384+0000", 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 11.385668552, 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092481160000000007, 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.912 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436384+0000", 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436384+0000", 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.913 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436385+0000", 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2799999999999995e-07 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436405+0000", 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0169e-05 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436443+0000", 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7920000000000003e-05 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.914 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436451+0000", 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4120000000000001e-06 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436507+0000", 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5512e-05 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.436516+0000", 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2699999999999993e-06 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.915 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444738+0000", 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082217950000000005 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.445632+0000", 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089431000000000005 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.916 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=188, reqid=client.4440.0:379, at_version=47'56, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 6.3698120410000003, 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011758437, 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.917 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452266+0000", 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4151e-05 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.918 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459768+0000", 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075025179999999997 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459778+0000", 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0244999999999999e-05 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459835+0000", 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6394999999999998e-05 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.919 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459845+0000", 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0518e-05 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463963+0000", 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041182450000000004 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463999+0000", 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5667999999999998e-05 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.920 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 6.357624468, 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:28.921 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.922 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.924 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:426 3.13s0 3:ce079411:::smithi01639990-2:d [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 1.3145521419999999, 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097940059999999992, 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:28.925 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 426 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:28.926 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4799999999999999e-07 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507505+0000", 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0149999999999997e-06 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:28.927 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507511+0000", 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4540000000000004e-06 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507517+0000", 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0749999999999997e-06 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.508922+0000", 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001405211 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.928 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.509012+0000", 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9776000000000002e-05 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.517295+0000", 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082831269999999995 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:28.929 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:28.931 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:28.931 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:28.928+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:29.031 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:29.029+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:29.039 INFO:tasks.rados.rados.0.smithi016.stdout:184: finishing write tid 4 to smithi01639990-37 2024-09-16T04:08:29.041 INFO:tasks.rados.rados.0.smithi016.stdout:184: finishing write tid 5 to smithi01639990-37 2024-09-16T04:08:29.041 INFO:tasks.rados.rados.0.smithi016.stdout:184: oid 37 updating version 0 to 61 2024-09-16T04:08:29.041 INFO:tasks.rados.rados.0.smithi016.stdout:184: oid 37 version 61 is already newer than 60 2024-09-16T04:08:29.041 INFO:tasks.rados.rados.0.smithi016.stdout:184: oid 37 version 61 is already newer than 59 2024-09-16T04:08:29.041 INFO:tasks.rados.rados.0.smithi016.stdout:184: oid 37 version 61 is already newer than 58 2024-09-16T04:08:29.041 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 37 v 61 (ObjNum 95 snap 12 seq_num 95) dirty exists 2024-09-16T04:08:29.041 INFO:tasks.rados.rados.0.smithi016.stdout:184: left oid 37 (ObjNum 95 snap 12 seq_num 95) 2024-09-16T04:08:29.042 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:29.043 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:29.043 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:29.043 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:29.052 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:29.132 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:29.130+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:29.154 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:29.155 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:29.155 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:29.155 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:29.155 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:29.163 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:29.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:29.183 INFO:tasks.rados.rados.0.smithi016.stdout:182: finishing write tid 4 to smithi01639990-47 2024-09-16T04:08:29.184 INFO:tasks.rados.rados.0.smithi016.stdout:182: finishing write tid 5 to smithi01639990-47 2024-09-16T04:08:29.184 INFO:tasks.rados.rados.0.smithi016.stdout:182: oid 47 updating version 0 to 45 2024-09-16T04:08:29.184 INFO:tasks.rados.rados.0.smithi016.stdout:182: oid 47 updating version 45 to 46 2024-09-16T04:08:29.185 INFO:tasks.rados.rados.0.smithi016.stdout:182: oid 47 updating version 46 to 47 2024-09-16T04:08:29.185 INFO:tasks.rados.rados.0.smithi016.stdout:182: oid 47 version 47 is already newer than 44 2024-09-16T04:08:29.185 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 47 v 47 (ObjNum 94 snap 12 seq_num 94) dirty exists 2024-09-16T04:08:29.185 INFO:tasks.rados.rados.0.smithi016.stdout:182: left oid 47 (ObjNum 94 snap 12 seq_num 94) 2024-09-16T04:08:29.233 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:29.231+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.162598908, 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073296819999999997, 2024-09-16T04:08:29.269 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103494+0000", 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1399999999999997e-07 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103514+0000", 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9698999999999999e-05 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103592+0000", 2024-09-16T04:08:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8378000000000001e-05 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103598+0000", 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9100000000000002e-06 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105790+0000", 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021916359999999998 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105798+0000", 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2490000000000001e-06 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110788+0000", 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049902660000000001 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110823+0000", 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5129999999999997e-05 2024-09-16T04:08:29.273 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=34, reqid=client.4440.0:211, at_version=33'20, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.150381972000002, 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078650630000000003, 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115711+0000", 2024-09-16T04:08:29.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2399999999999998e-07 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115771+0000", 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9478999999999998e-05 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119749+0000", 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039784099999999999 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119760+0000", 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0664e-05 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119899+0000", 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013905300000000001 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119919+0000", 2024-09-16T04:08:29.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9854999999999999e-05 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123544+0000", 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036256499999999998 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123575+0000", 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1328000000000002e-05 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.278 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=35, reqid=client.4440.0:212, at_version=33'21, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.147626850000002, 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077855709999999998, 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:29.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118551+0000", 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4878999999999998e-05 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120595+0000", 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020432639999999999 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120604+0000", 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0250000000000008e-06 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122166+0000", 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015625599999999999 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122175+0000", 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8470000000000002e-06 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126217+0000", 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040415249999999998 2024-09-16T04:08:29.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126251+0000", 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4570999999999998e-05 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.124399087, 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:29.283 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:29.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:29.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:296 3.9s0 3:9fd10cc9:::smithi01639990-50:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.161519197000001, 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010626313, 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.289 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104575+0000", 2024-09-16T04:08:29.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1349999999999999e-06 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104586+0000", 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0603e-05 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104606+0000", 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9840999999999999e-05 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104615+0000", 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0189999999999995e-06 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104674+0000", 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9534999999999998e-05 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104680+0000", 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9320000000000001e-06 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109134+0000", 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044534120000000003 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109195+0000", 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0983000000000002e-05 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109711+0000", 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051621600000000001 2024-09-16T04:08:29.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.110753+0000", 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001042533 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115115+0000", 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043613030000000004 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115200+0000", 2024-09-16T04:08:29.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4801000000000003e-05 2024-09-16T04:08:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.295 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.817069826000001, 2024-09-16T04:08:29.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085197069999999996, 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4000000000000001e-07 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449160+0000", 2024-09-16T04:08:29.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013637999999999999 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451237+0000", 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020768179999999998 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451246+0000", 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451532+0000", 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00028571800000000002 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451542+0000", 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0118999999999999e-05 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457500+0000", 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059576680000000002 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457542+0000", 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2716999999999998e-05 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.806343479000001, 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:08:29.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:08:29.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.67580132, 2024-09-16T04:08:29.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:08:29.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:08:29.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:363 3.9s0 3:94de6fe1:::smithi01639990-3:head [stat,delete] snapc 8=[6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.826234660000001, 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085012569999999999, 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 363 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439860+0000", 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439869+0000", 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3170000000000008e-06 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439888+0000", 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9293e-05 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439897+0000", 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439953+0000", 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6317999999999998e-05 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439959+0000", 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3040000000000004e-06 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441596+0000", 2024-09-16T04:08:29.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001637655 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441659+0000", 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2658999999999998e-05 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.442297+0000", 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063811800000000002 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.443562+0000", 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001265361 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448304+0000", 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047413560000000004 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448359+0000", 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5189000000000003e-05 2024-09-16T04:08:29.318 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.594878767, 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:29.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.4344747089999998, 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:08:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:08:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:08:29.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:08:29.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:419 3.9s0 3:94de6fe1:::smithi01639990-3:head [append 0~786432] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:29.329 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.7635806349999998, 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082147929999999997, 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 419 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.330 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502514+0000", 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.65e-06 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503628+0000", 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001113239 2024-09-16T04:08:29.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503646+0000", 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7906000000000001e-05 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503653+0000", 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2339999999999997e-06 2024-09-16T04:08:29.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503741+0000", 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8436999999999994e-05 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503820+0000", 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8751000000000005e-05 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505036+0000", 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001216017 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510097+0000", 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050614550000000003 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:29.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510625+0000", 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000527667 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510727+0000", 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101437 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:420 3.9s0 3:94de6fe1:::smithi01639990-3:head [append 0~491520] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.7609487339999998, 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089351980000000001, 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.337 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 420 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:29.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505614+0000", 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00046960099999999999 2024-09-16T04:08:29.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505627+0000", 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3275000000000001e-05 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505636+0000", 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2609999999999993e-06 2024-09-16T04:08:29.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505739+0000", 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010322700000000001 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505847+0000", 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107929 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.506958+0000", 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011114580000000001 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510232+0000", 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032740080000000001 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.513941+0000", 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037089470000000002 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514079+0000", 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000137744 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:422 3.9s0 3:94de6fe1:::smithi01639990-3:head [setxattr _header (56) in=7b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507590+0000", 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.758502499, 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007312804, 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.343 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 422 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507590+0000", 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507590+0000", 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507591+0000", 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507594+0000", 2024-09-16T04:08:29.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5329999999999999e-06 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507602+0000", 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5379999999999997e-06 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507607+0000", 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4500000000000003e-06 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.509353+0000", 2024-09-16T04:08:29.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001745468 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.509454+0000", 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101718 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.509859+0000", 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040448799999999999 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.513811+0000", 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039521690000000002 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514806+0000", 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099465199999999995 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514903+0000", 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6979999999999996e-05 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.348 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:423 3.9s0 3:94de6fe1:::smithi01639990-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.758476358, 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010412819, 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.349 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 423 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507617+0000", 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6599999999999996e-07 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507621+0000", 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9609999999999999e-06 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507628+0000", 2024-09-16T04:08:29.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3139999999999998e-06 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507637+0000", 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1840000000000005e-06 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510428+0000", 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027909060000000001 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510505+0000", 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6679000000000002e-05 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514954+0000", 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044494410000000002 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515004+0000", 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0244999999999998e-05 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518029+0000", 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030245229999999999 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:427 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507644+0000", 2024-09-16T04:08:29.354 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.7584480440000001, 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079136850000000002, 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 427 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507644+0000", 2024-09-16T04:08:29.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507644+0000", 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507645+0000", 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4499999999999997e-07 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507649+0000", 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5789999999999997e-06 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507655+0000", 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6729999999999997e-06 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507660+0000", 2024-09-16T04:08:29.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0329999999999999e-06 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510813+0000", 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031532259999999999 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510938+0000", 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000124757 2024-09-16T04:08:29.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.511503+0000", 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00056476499999999995 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514248+0000", 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002745272 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:29.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515498+0000", 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012494139999999999 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515558+0000", 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0520999999999999e-05 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:428 3.9s0 3:92dcccbe:::smithi01639990-16:head [rollback c] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.7584139730000001, 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0081187840000000004, 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.361 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.362 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.362 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 428 2024-09-16T04:08:29.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.362 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.362 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.362 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:29.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507679+0000", 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0900000000000002e-07 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507684+0000", 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3199999999999999e-06 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507691+0000", 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1160000000000001e-06 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507697+0000", 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3129999999999998e-06 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512167+0000", 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044708880000000001 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512254+0000", 2024-09-16T04:08:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6329999999999995e-05 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512662+0000", 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040792999999999998 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514513+0000", 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018516019999999999 2024-09-16T04:08:29.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515756+0000", 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001242976 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515797+0000", 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0899999999999998e-05 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.7583698719999998, 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:29.368 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:08:29.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:08:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:29.374 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.736656847, 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:08:29.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:29.380 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.73660462000000004, 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:29.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:08:29.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.385 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.386 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:29.386 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:29.386 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:29.387 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:29.387 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:29.388 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:29.388 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:29.389 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:29.389 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:29.386+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:29.390 INFO:tasks.rados.rados.0.smithi016.stdout:186: finishing rollback tid 1 to smithi01639990-23 2024-09-16T04:08:29.390 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 23 v 0 (ObjNum 33 snap 0 seq_num 2168494960) dirty dne 2024-09-16T04:08:29.390 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:29.391 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:29.399 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:29.421 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:29.443 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:29.464 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:29.464 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:29.486 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:29.486 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:182: done (7 left) 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:184: done (6 left) 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:186: done (5 left) 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:188: done (4 left) 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:189: done (3 left) 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:190: done (2 left) 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:192: done (1 left) 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:193: done (0 left) 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:194: copy_from oid 22 from oid 42 current snap is 13 2024-09-16T04:08:29.487 INFO:tasks.rados.rados.0.smithi016.stdout:195: snap_create 2024-09-16T04:08:29.488 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:29.486+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:29.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:08:29.502 INFO:tasks.rados.rados.0.smithi016.stdout:194: finishing copy_from racing read to smithi01639990-22 2024-09-16T04:08:29.506 INFO:tasks.rados.rados.0.smithi016.stdout:194: finishing copy_from to smithi01639990-22 2024-09-16T04:08:29.506 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 22 v 27 (ObjNum 91 snap 12 seq_num 91) dirty exists 2024-09-16T04:08:29.510 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:29.510 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:29.532 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:29.533 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:29.537 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:29.537 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:29.546 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:29.553 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:29.553 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:29.589 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:29.587+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:29.650 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:29.650 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:29.659 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:29.690 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:29.687+0000 7f8f3790a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33068) UID: 0 2024-09-16T04:08:29.762 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:29.762 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:29.770 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T04:08:29.770 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:08:29.770 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-16T04:08:29.771 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T04:08:29.771 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:08:29.771 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:08:29.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:29.791 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:29.788+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:29.797 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:08:29.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-16T04:08:29.877 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:29.877 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:29.885 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:29.892 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:29.890+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:29.988 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:29.988 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:29.992 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:29.990+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:29.997 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:30.093 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:30.092+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:30.122 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:30.122 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:30.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:30.140 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-09-16T04:08:30.137+0000 7f8c5c132640 -1 mon.a@0(leader).osd e54 definitely_dead 0 2024-09-16T04:08:30.195 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:30.194+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:30.229 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:30.230 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:30.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:30.296 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:30.295+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:30.386 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:30.386 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:30.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:30.397 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:30.395+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-16T04:08:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:marked down osd.1. 2024-09-16T04:08:30.434 INFO:tasks.rados.rados.0.smithi016.stdout:194: done (1 left) 2024-09-16T04:08:30.434 INFO:tasks.rados.rados.0.smithi016.stdout:195: done (0 left) 2024-09-16T04:08:30.434 INFO:tasks.rados.rados.0.smithi016.stdout:196: snap_create 2024-09-16T04:08:30.459 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-16T04:08:30.459 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-09-16T04:08:30.501 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:30.498+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:30.513 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:30.513 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:30.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:30.599 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:30.598+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:30.672 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:30.672 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:30.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:30.725 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:30.722+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:30.789 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:30.789 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:30.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:30.800 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:30.798+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:30.810 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-09-16T04:08:30.807+0000 7f8c5c132640 -1 mon.a@0(leader).osd e55 definitely_dead 0 2024-09-16T04:08:30.901 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:30.900+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:30.916 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:30.916 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:30.925 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:31.002 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:30.999+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:31.032 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:31.032 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:31.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:31.102 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:31.099+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:31.148 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:31.148 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:31.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:31.241 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:31.238+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:31.305 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:31.305 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:31.305 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:31.303+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:31.314 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:31.405 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:31.403+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:31.430 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:31.431 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:31.432 INFO:teuthology.orchestra.run.smithi016.stderr:marked out osd.1. 2024-09-16T04:08:31.437 INFO:tasks.rados.rados.0.smithi016.stdout:196: done (0 left) 2024-09-16T04:08:31.438 INFO:tasks.rados.rados.0.smithi016.stdout:197: setattr oid 41 current snap is 15 2024-09-16T04:08:31.439 INFO:tasks.rados.rados.0.smithi016.stdout:198: read oid 13 snap -1 2024-09-16T04:08:31.439 INFO:tasks.rados.rados.0.smithi016.stdout:198: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-16T04:08:31.439 INFO:tasks.rados.rados.0.smithi016.stdout:199: copy_from oid 29 from oid 24 current snap is 15 2024-09-16T04:08:31.439 INFO:tasks.rados.rados.0.smithi016.stdout:200: rollback oid 7 current snap is 15 2024-09-16T04:08:31.439 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 7 to 9 2024-09-16T04:08:31.439 INFO:tasks.rados.rados.0.smithi016.stdout:201: rollback oid 6 current snap is 15 2024-09-16T04:08:31.439 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 6 to 14 2024-09-16T04:08:31.439 INFO:tasks.rados.rados.0.smithi016.stdout:202: snap_remove snap 11 2024-09-16T04:08:31.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:31.456 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-09-16T04:08:31.456 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:08:31.456 DEBUG:teuthology.orchestra.run.smithi016:> type ceph-objectstore-tool 2024-09-16T04:08:31.471 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T04:08:31.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-16T04:08:31.513 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:31.511+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:31.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:31.588 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:31.589 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:31.597 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:31.609 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:31.606+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:31.697 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:31.698 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:31.698 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T04:08:31.700 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:31.701 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:31.708 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:31.706+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:31.709 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:31.718 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:31.718 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:31.739 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:31.739 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:31.761 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:31.762 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:31.782 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:31.782 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:31.805 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:31.805 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:31.808 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:31.805+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:31.815 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:31.815 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:31.816 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:31.816 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:31.825 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:31.826 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:31.826 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:31.848 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:31.848 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:31.870 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:31.891 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:31.910 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:31.907+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:31.913 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:08:31.917 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:08:31.929 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:31.929 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:31.929 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:31.929 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:31.929 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:31.938 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:32.022 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:32.020+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:32.082 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:32.082 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 33/27 ECSubWrite(tid=37, reqid=osd.8.0:37, at_version=33'25, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.980975725, 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088041119999999994, 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.083 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098707+0000", 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7000000000000001e-07 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098728+0000", 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1432000000000001e-05 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098770+0000", 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1486999999999998e-05 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098780+0000", 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7280000000000007e-06 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098852+0000", 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2461999999999998e-05 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.098862+0000", 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9369999999999999e-06 2024-09-16T04:08:32.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107472+0000", 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086102990000000001 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107511+0000", 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8497000000000001e-05 2024-09-16T04:08:32.087 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.088 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.088 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.088 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:32.088 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:32.088 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.979805113000001, 2024-09-16T04:08:32.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011088002, 2024-09-16T04:08:32.088 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099877+0000", 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099878+0000", 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8400000000000003e-07 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099902+0000", 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3629999999999999e-05 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099934+0000", 2024-09-16T04:08:32.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2388999999999998e-05 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099943+0000", 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4800000000000001e-06 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.100212+0000", 2024-09-16T04:08:32.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026930999999999998 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.100220+0000", 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8339999999999999e-06 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110928+0000", 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010708333 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110965+0000", 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7141999999999998e-05 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s7 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.977592792999999, 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088092050000000005, 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.093 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102090+0000", 2024-09-16T04:08:32.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.18e-07 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102106+0000", 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6116999999999998e-05 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102134+0000", 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7963e-05 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102140+0000", 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102224+0000", 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3634000000000006e-05 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.102233+0000", 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1609999999999992e-06 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110868+0000", 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0086352040000000005 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110899+0000", 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0852999999999998e-05 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:32.097 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.976666957999999, 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013925085, 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103016+0000", 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5499999999999999e-07 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103038+0000", 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2416e-05 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103157+0000", 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000118439 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103166+0000", 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9819999999999997e-06 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103225+0000", 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8996000000000002e-05 2024-09-16T04:08:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103235+0000", 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9909999999999997e-06 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116899+0000", 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013664549999999999 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116941+0000", 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1455999999999997e-05 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.976641657000002, 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013929731000000001, 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.102 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5899999999999998e-07 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103057+0000", 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5778e-05 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103193+0000", 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0001359 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103199+0000", 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1279999999999996e-06 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107717+0000", 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045183769999999996 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107726+0000", 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5420000000000005e-06 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116950+0000", 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092240229999999996 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116970+0000", 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0724e-05 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.106 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s5 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.972777966999999, 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0080941920000000001, 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106904+0000", 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106905+0000", 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4999999999999999e-07 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106926+0000", 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1202000000000001e-05 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107015+0000", 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9407999999999995e-05 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107034+0000", 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8896999999999999e-05 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107499+0000", 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00046474800000000002 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107509+0000", 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0390999999999999e-05 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114906+0000", 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073969409999999998 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.114999+0000", 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2454999999999995e-05 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds7 37/31 ECSubWrite(tid=163, reqid=osd.7.0:163, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.971124763999999, 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084527260000000007, 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.111 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108558+0000", 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0100000000000004e-07 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108578+0000", 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9366999999999999e-05 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108616+0000", 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8056999999999998e-05 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108624+0000", 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.824e-06 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108684+0000", 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0019999999999998e-05 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.108693+0000", 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2599999999999994e-06 2024-09-16T04:08:32.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.116974+0000", 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082815990000000006 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.117010+0000", 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5898000000000001e-05 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds7 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.97015086, 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091377019999999993, 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109532+0000", 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6499999999999998e-07 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109554+0000", 2024-09-16T04:08:32.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2030000000000001e-05 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109587+0000", 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3019999999999999e-05 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.109603+0000", 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5678000000000001e-05 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110514+0000", 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00091074700000000005 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110523+0000", 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4639999999999995e-06 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118636+0000", 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008112157 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.118669+0000", 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3741000000000001e-05 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.635210441, 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011544518, 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.120 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444473+0000", 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4499999999999996e-07 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447203+0000", 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002729834 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447277+0000", 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4078999999999993e-05 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447283+0000", 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4169999999999997e-06 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447458+0000", 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00017450300000000001 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447469+0000", 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1062e-05 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.455978+0000", 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085090730000000007 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456017+0000", 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8804999999999999e-05 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:352 3.cs0 3:338c6e39:::smithi01639990-33:6 [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.633198583, 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078540529999999997, 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 352 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446484+0000", 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9200000000000001e-07 2024-09-16T04:08:32.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446487+0000", 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0489999999999999e-06 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446494+0000", 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5180000000000001e-06 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.446499+0000", 2024-09-16T04:08:32.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.9010000000000002e-06 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447665+0000", 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0011659120000000001 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447747+0000", 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2785000000000001e-05 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.454338+0000", 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065903960000000001 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/27 ECSubWriteReply(tid=88, last_complete=42'39, committed=1, applied=1))", 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.631236424000001, 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49630058100000002, 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448447+0000", 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448451+0000", 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3030000000000001e-06 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944646+0000", 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49619510999999999 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944655+0000", 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8480000000000007e-06 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944717+0000", 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2065999999999998e-05 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944747+0000", 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9505999999999998e-05 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.132 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.450032+0000", 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 14.629649949000001, 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088454970000000008, 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450032+0000", 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450032+0000", 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450033+0000", 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2399999999999998e-07 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451065+0000", 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0010323089999999999 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451134+0000", 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8096999999999995e-05 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451140+0000", 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5939999999999999e-06 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451197+0000", 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6665e-05 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.451206+0000", 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3530000000000001e-06 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.458843+0000", 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0076371549999999996 2024-09-16T04:08:32.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.458878+0000", 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4700000000000003e-05 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDScrubReserve(3.0s7,QREQUEST,e:47,from:11(0),reservation_nonce:4)", 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.0525146099999994, 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17961150300000001, 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.137 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027169+0000", 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027173+0000", 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1300000000000003e-06 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206636+0000", 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17946284700000001 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206644+0000", 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9650000000000008e-06 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206712+0000", 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8215999999999999e-05 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206735+0000", 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2789000000000002e-05 2024-09-16T04:08:32.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206779+0000", 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4669000000000002e-05 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:414 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~770048] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5951403129999999, 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012099706, 2024-09-16T04:08:32.141 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 414 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484544+0000", 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485808+0000", 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012637099999999999 2024-09-16T04:08:32.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485825+0000", 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7722000000000001e-05 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485834+0000", 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9749999999999996e-06 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485905+0000", 2024-09-16T04:08:32.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1131999999999996e-05 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.486012+0000", 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000106214 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487551+0000", 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015389830000000001 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.489803+0000", 2024-09-16T04:08:32.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0022524839999999999 2024-09-16T04:08:32.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:32.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496554+0000", 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067513879999999997 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496642+0000", 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7385000000000004e-05 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:415 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~475136] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.487079+0000", 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5926038380000005, 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097770519999999996, 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 415 2024-09-16T04:08:32.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487079+0000", 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487079+0000", 2024-09-16T04:08:32.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487079+0000", 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6199999999999998e-07 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487619+0000", 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053991599999999999 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487633+0000", 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4158000000000001e-05 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487654+0000", 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0669000000000001e-05 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.488619+0000", 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00096477599999999998 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.488718+0000", 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9809000000000006e-05 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.489445+0000", 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072619099999999995 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490031+0000", 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000586514 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496790+0000", 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.006758633 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496856+0000", 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5823999999999999e-05 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.153 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:416 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~294912 in=294912b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5890599019999998, 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16418209, 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 416 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490624+0000", 2024-09-16T04:08:32.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3069999999999999e-06 2024-09-16T04:08:32.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490921+0000", 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00029678400000000002 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490932+0000", 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1239e-05 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490938+0000", 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0379999999999996e-06 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490980+0000", 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2401999999999998e-05 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491037+0000", 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6768000000000003e-05 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491591+0000", 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055345000000000004 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492749+0000", 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0011580970000000001 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654745+0000", 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16199656800000001 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654805+0000", 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9437e-05 2024-09-16T04:08:32.159 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:417 3.cs0 3:38c68ed3:::smithi01639990-42:head [setxattr _header (57) in=64b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5887298830000001, 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67294876800000003, 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.160 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 417 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2699999999999999e-07 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490956+0000", 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0130000000000002e-06 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490961+0000", 2024-09-16T04:08:32.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1510000000000003e-06 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490966+0000", 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7960000000000003e-06 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491934+0000", 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00096816399999999998 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492020+0000", 2024-09-16T04:08:32.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5827000000000003e-05 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492479+0000", 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045912299999999997 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492973+0000", 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049418400000000005 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163802+0000", 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67082863599999998 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163901+0000", 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9246999999999994e-05 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:418 3.cs0 3:38c68ed3:::smithi01639990-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:32.165 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5887087040000001, 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67426341400000001, 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 418 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:08:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8000000000000001e-07 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490976+0000", 2024-09-16T04:08:32.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.973e-06 2024-09-16T04:08:32.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490983+0000", 2024-09-16T04:08:32.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4699999999999999e-06 2024-09-16T04:08:32.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490988+0000", 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5509999999999999e-06 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493081+0000", 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020931029999999998 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493165+0000", 2024-09-16T04:08:32.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3988999999999998e-05 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164330+0000", 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67116498999999996 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164423+0000", 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000004e-05 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.165237+0000", 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081446800000000005 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 52/27 ECSubWriteReply(tid=99, last_complete=52'26, committed=1, applied=1))", 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5867670130000002, 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16189895200000001, 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492916+0000", 2024-09-16T04:08:32.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1699999999999998e-07 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492919+0000", 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6970000000000002e-06 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654619+0000", 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16170007 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654628+0000", 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9080000000000001e-06 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654685+0000", 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7349999999999998e-05 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654814+0000", 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012941 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 52/27 ECSubWriteReply(tid=100, last_complete=52'27, committed=1, applied=1))", 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.586324941, 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67093597800000004, 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493358+0000", 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493360+0000", 2024-09-16T04:08:32.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2019999999999998e-06 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163618+0000", 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67025769000000002 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163644+0000", 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6089999999999999e-05 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163713+0000", 2024-09-16T04:08:32.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9326999999999995e-05 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164293+0000", 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00058030099999999997 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.178 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:32.180 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:32.180 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:32.178+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:32.280 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:32.278+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:32.322 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:32.322 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:32.322 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:32.322 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:32.330 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:32.381 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.378+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:32.435 INFO:tasks.rados.rados.0.smithi016.stdout:202: done (5 left) 2024-09-16T04:08:32.435 INFO:tasks.rados.rados.0.smithi016.stdout:203: snap_create 2024-09-16T04:08:32.436 INFO:tasks.rados.rados.0.smithi016.stdout:198: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-16T04:08:32.441 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.441 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.441 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.441 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.441 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.441 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.442 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.442 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.442 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.442 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.438+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.449 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.438+0000 7f650f91d640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.449 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.445+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.449 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.445+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.449 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.445+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.449 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.445+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.449 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:32.446+0000 7f304f68e640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,2,3,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.451 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.438+0000 7f650f91d640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.451 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.439+0000 7f650f91d640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.452 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.439+0000 7f650f91d640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.454 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.448+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.448+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.449+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.449+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.452+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.452+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.452+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.452+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.452+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.452+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.452+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.455 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.452+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.456 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.453+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.456 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.453+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.456 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.453+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.456 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.453+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.456 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.453+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.456 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.453+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.456 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.453+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.456 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:32.453+0000 7f650b915640 -1 ErasureCodeLrc: _minimum_to_decode not enough chunks in 0,1,4,5,7 to read 0,1,2,3,4,5,6,7 2024-09-16T04:08:32.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:08:32.489 INFO:tasks.rados.rados.0.smithi016.stdout:200: finishing rollback tid 0 to smithi01639990-7 2024-09-16T04:08:32.489 INFO:tasks.rados.rados.0.smithi016.stdout:201: finishing rollback tid 0 to smithi01639990-6 2024-09-16T04:08:32.490 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:32.486+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:32.490 INFO:tasks.rados.rados.0.smithi016.stdout:200: finishing rollback tid 1 to smithi01639990-7 2024-09-16T04:08:32.490 INFO:tasks.rados.rados.0.smithi016.stdout:200: finishing rollback tid 2 to smithi01639990-7 2024-09-16T04:08:32.490 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 7 v 21 (ObjNum 64 snap 2 seq_num 64) dirty exists 2024-09-16T04:08:32.491 INFO:tasks.rados.rados.0.smithi016.stdout:201: finishing rollback tid 1 to smithi01639990-6 2024-09-16T04:08:32.497 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 41 v 58 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-09-16T04:08:32.502 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:32.502 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:32.502 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:32.502 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:32.502 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:32.508 INFO:tasks.rados.rados.0.smithi016.stdout:199: finishing copy_from racing read to smithi01639990-29 2024-09-16T04:08:32.509 INFO:tasks.rados.rados.0.smithi016.stdout:199: finishing copy_from to smithi01639990-29 2024-09-16T04:08:32.510 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 29 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T04:08:32.515 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:32.527 INFO:tasks.rados.rados.0.smithi016.stdout:201: finishing rollback tid 2 to smithi01639990-6 2024-09-16T04:08:32.527 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 6 v 27 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T04:08:32.584 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:32.581+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:32.640 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.534362114, 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0068480019999999997, 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.641 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103704+0000", 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7100000000000001e-07 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103727+0000", 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3694999999999999e-05 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103810+0000", 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3091999999999997e-05 2024-09-16T04:08:32.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103816+0000", 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4519999999999996e-06 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103971+0000", 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000154993 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103982+0000", 2024-09-16T04:08:32.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0933e-05 2024-09-16T04:08:32.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110491+0000", 2024-09-16T04:08:32.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065096800000000003 2024-09-16T04:08:32.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110551+0000", 2024-09-16T04:08:32.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9985999999999998e-05 2024-09-16T04:08:32.647 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.647 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.647 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.647 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:32.648 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:32.648 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.534312776, 2024-09-16T04:08:32.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067325809999999996, 2024-09-16T04:08:32.648 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.648 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.648 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.649 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:32.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:32.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103754+0000", 2024-09-16T04:08:32.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5899999999999995e-07 2024-09-16T04:08:32.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103768+0000", 2024-09-16T04:08:32.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4562999999999999e-05 2024-09-16T04:08:32.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103805+0000", 2024-09-16T04:08:32.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6730000000000002e-05 2024-09-16T04:08:32.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103813+0000", 2024-09-16T04:08:32.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:08:32.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103868+0000", 2024-09-16T04:08:32.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4475000000000003e-05 2024-09-16T04:08:32.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103877+0000", 2024-09-16T04:08:32.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7099999999999996e-06 2024-09-16T04:08:32.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110444+0000", 2024-09-16T04:08:32.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065675170000000001 2024-09-16T04:08:32.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110485+0000", 2024-09-16T04:08:32.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1177000000000002e-05 2024-09-16T04:08:32.657 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.657 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.658 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.658 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/27 ECSubWrite(tid=124, reqid=osd.3.0:124, at_version=37'22, trim_to=0'0, roll_forward_to=35'20))", 2024-09-16T04:08:32.658 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:32.658 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.534728476000002, 2024-09-16T04:08:32.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070925160000000001, 2024-09-16T04:08:32.659 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.659 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.659 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.659 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:32.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:32.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:32.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6999999999999999e-07 2024-09-16T04:08:32.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103359+0000", 2024-09-16T04:08:32.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1835999999999999e-05 2024-09-16T04:08:32.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103683+0000", 2024-09-16T04:08:32.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00032427399999999999 2024-09-16T04:08:32.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103693+0000", 2024-09-16T04:08:32.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7380000000000006e-06 2024-09-16T04:08:32.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103756+0000", 2024-09-16T04:08:32.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3115999999999997e-05 2024-09-16T04:08:32.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103766+0000", 2024-09-16T04:08:32.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0043e-05 2024-09-16T04:08:32.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110407+0000", 2024-09-16T04:08:32.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0066407230000000003 2024-09-16T04:08:32.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110430+0000", 2024-09-16T04:08:32.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2616000000000002e-05 2024-09-16T04:08:32.668 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.668 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.669 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.669 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.533898203, 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008202661, 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104168+0000", 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0200000000000001e-07 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104188+0000", 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9933999999999999e-05 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104232+0000", 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4196000000000002e-05 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104241+0000", 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7979999999999995e-06 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104928+0000", 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687124 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104937+0000", 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5269999999999998e-06 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112328+0000", 2024-09-16T04:08:32.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073914510000000003 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112370+0000", 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1928999999999998e-05 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.531178595, 2024-09-16T04:08:32.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082659529999999995, 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1600000000000002e-07 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106910+0000", 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2954000000000001e-05 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106990+0000", 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9788000000000002e-05 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106997+0000", 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4379999999999998e-06 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107267+0000", 2024-09-16T04:08:32.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026999599999999999 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107276+0000", 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9530000000000005e-06 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115126+0000", 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078491940000000003 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115153+0000", 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7214000000000001e-05 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.18812537, 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091491550000000008, 2024-09-16T04:08:32.679 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5800000000000001e-07 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450512+0000", 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057160000000000002 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450541+0000", 2024-09-16T04:08:32.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8835000000000001e-05 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450549+0000", 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3080000000000008e-06 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450607+0000", 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7763999999999998e-05 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450616+0000", 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459045+0000", 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084290709999999998 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459089+0000", 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4233000000000002e-05 2024-09-16T04:08:32.683 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=160, reqid=client.4440.0:347, at_version=42'32, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.185503288, 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065520760000000004, 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.684 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452563+0000", 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6300000000000005e-07 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452621+0000", 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8488e-05 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452654+0000", 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2750999999999999e-05 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452659+0000", 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.293e-06 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452730+0000", 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0173999999999996e-05 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452738+0000", 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6260000000000001e-06 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459095+0000", 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0063566530000000003 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459114+0000", 2024-09-16T04:08:32.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9428000000000001e-05 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=161, reqid=client.4440.0:348, at_version=42'33, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.185440681999999, 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065540390000000002, 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.689 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6300000000000001e-07 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452632+0000", 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6699999999999997e-06 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452678+0000", 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5760000000000002e-05 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452682+0000", 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.036e-06 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453280+0000", 2024-09-16T04:08:32.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059861699999999996 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453289+0000", 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9590000000000001e-06 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459117+0000", 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058281770000000004 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459179+0000", 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1556999999999994e-05 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:469 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~491520] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0852924059999998, 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098323030000000006, 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.694 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 469 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552774+0000", 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1999999999999999e-06 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553284+0000", 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00050997900000000003 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553297+0000", 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2723e-05 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553306+0000", 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5969999999999992e-06 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553363+0000", 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6922999999999997e-05 2024-09-16T04:08:32.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553436+0000", 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3090000000000007e-05 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.554087+0000", 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065173499999999997 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.558220+0000", 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041326330000000001 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562517+0000", 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042965859999999998 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562605+0000", 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8837000000000003e-05 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.699 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:470 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~589824] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.556072+0000", 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0819938740000001, 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079188939999999992, 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.700 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 470 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556072+0000", 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556072+0000", 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556073+0000", 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0729999999999999e-06 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556674+0000", 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00060172999999999997 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556685+0000", 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0111e-05 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556689+0000", 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8420000000000004e-06 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556732+0000", 2024-09-16T04:08:32.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2654000000000002e-05 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556832+0000", 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9953999999999999e-05 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.557727+0000", 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089499499999999999 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562616+0000", 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048893870000000002 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563944+0000", 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013275380000000001 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563991+0000", 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6610000000000003e-05 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.705 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:471 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~557056] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.559406+0000", 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0786593120000001, 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0071509360000000001, 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.706 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 471 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.559406+0000", 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.559406+0000", 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.559407+0000", 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2780000000000001e-06 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560034+0000", 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00062697999999999996 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560044+0000", 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:08:32.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560050+0000", 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1059999999999996e-06 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560093+0000", 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2673000000000003e-05 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560217+0000", 2024-09-16T04:08:32.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124175 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.561108+0000", 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089074 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566027+0000", 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049195159999999996 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:32.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566499+0000", 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000472032 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566557+0000", 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7989999999999999e-05 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:472 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~16384] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:32.711 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.560062+0000", 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0780035010000004, 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067563500000000004, 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 472 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560062+0000", 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560062+0000", 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560062+0000", 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8299999999999997e-07 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560072+0000", 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8539999999999992e-06 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560076+0000", 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1459999999999998e-06 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560080+0000", 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.168e-06 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.561729+0000", 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001649799 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.561798+0000", 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8555999999999999e-05 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562168+0000", 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037029100000000002 2024-09-16T04:08:32.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566620+0000", 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004451404 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566775+0000", 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015578599999999999 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566818+0000", 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2862999999999999e-05 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:473 3.as0 3:5e52341f:::smithi01639990-38:head [setxattr _header (57) in=7b] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.560085+0000", 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0779807200000002, 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084263350000000001, 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.717 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 473 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560085+0000", 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560085+0000", 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560085+0000", 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560087+0000", 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7880000000000001e-06 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560090+0000", 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3409999999999999e-06 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560093+0000", 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.672e-06 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562731+0000", 2024-09-16T04:08:32.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0026385169999999999 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562783+0000", 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1215999999999999e-05 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563136+0000", 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035292899999999998 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566622+0000", 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003486556 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568451+0000", 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0018289389999999999 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568511+0000", 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0096000000000002e-05 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.722 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:474 3.as0 3:5e52341f:::smithi01639990-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.0779604450000004, 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097686969999999998, 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.723 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 474 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560106+0000", 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.63e-07 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560109+0000", 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.303e-06 2024-09-16T04:08:32.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560116+0000", 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3860000000000001e-06 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560122+0000", 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4450000000000004e-06 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563686+0000", 2024-09-16T04:08:32.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035639740000000001 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563776+0000", 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0278000000000005e-05 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568854+0000", 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0050781680000000001 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:32.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568970+0000", 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011579899999999999 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.569874+0000", 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00090388100000000004 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:32.728 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.20032875, 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015912599999999999, 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437757+0000", 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9735e-05 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437829+0000", 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1605000000000003e-05 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437838+0000", 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442352+0000", 2024-09-16T04:08:32.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045136799999999999 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442364+0000", 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2642e-05 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453618+0000", 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011254128 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453649+0000", 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1016000000000003e-05 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.199290141, 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014897297, 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.733 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438776+0000", 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3099999999999997e-07 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438797+0000", 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0596000000000001e-05 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438836+0000", 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9684000000000002e-05 2024-09-16T04:08:32.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438845+0000", 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6479999999999992e-06 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442677+0000", 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038317239999999999 2024-09-16T04:08:32.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442685+0000", 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453654+0000", 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010968637 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453673+0000", 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8967999999999999e-05 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.19730019200000001, 2024-09-16T04:08:32.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012700487999999999, 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8900000000000001e-07 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440785+0000", 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9850000000000001e-05 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445485+0000", 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046992329999999997 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445495+0000", 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0506000000000001e-05 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445668+0000", 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000172905 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445679+0000", 2024-09-16T04:08:32.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453438+0000", 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077588930000000002 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453466+0000", 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8175e-05 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=110, reqid=osd.5.0:109, at_version=0'0, trim_to=0'0, roll_forward_to=57'31))", 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.17349574000000001, 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011518758, 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8e-07 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464577+0000", 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9940000000000003e-06 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472531+0000", 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079544330000000003 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472538+0000", 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7789999999999998e-06 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472599+0000", 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0841999999999998e-05 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472610+0000", 2024-09-16T04:08:32.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1185999999999999e-05 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476060+0000", 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034496650000000002 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476089+0000", 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8679000000000001e-05 2024-09-16T04:08:32.748 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=37, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.14999293799999999, 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.039596059000000003, 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.749 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1e-07 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488075+0000", 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8789999999999999e-06 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526262+0000", 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.038187556999999997 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526270+0000", 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3209999999999999e-06 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526424+0000", 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015469 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.527669+0000", 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012444019999999999 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=38, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:08:32.753 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.107500661, 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028484058, 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:08:32.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530566+0000", 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.85e-07 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530570+0000", 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7319999999999998e-06 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:32.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558866+0000", 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028295515 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558875+0000", 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7849999999999997e-06 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558966+0000", 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1611000000000001e-05 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:32.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:32.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.559049+0000", 2024-09-16T04:08:32.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.263e-05 2024-09-16T04:08:32.757 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.757 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.757 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.757 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:32.757 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:32.757 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:32.759 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:32.759 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:32.755+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:32.859 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:32.857+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:32.862 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:32.863 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:32.863 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:32.863 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:32.871 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:32.960 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:32.958+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:32.977 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:32.977 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:32.977 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:32.977 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:32.977 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:32.985 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:33.061 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:33.058+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:33.089 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.146292369999998, 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.090 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:33.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:33.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:33.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:33.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:33.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:33.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:33.097 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.163888127, 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:33.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:33.102 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.103 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 42.153525621, 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:33.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:33.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:33.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:33.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:33.108 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.656937569, 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:33.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:33.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:33.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.202388984999999, 2024-09-16T04:08:33.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094233230000000008, 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 162 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884627+0000", 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884629+0000", 2024-09-16T04:08:33.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.187e-06 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884632+0000", 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.501e-06 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884635+0000", 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7389999999999999e-06 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892223+0000", 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075881029999999997 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892291+0000", 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7508000000000005e-05 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892412+0000", 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012124 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893979+0000", 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015676069999999999 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893987+0000", 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.306e-06 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:33.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894013+0000", 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6305999999999999e-05 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894050+0000", 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6492999999999998e-05 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.202376614000002, 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095013200000000006, 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 163 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2100000000000001e-07 2024-09-16T04:08:33.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884642+0000", 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.478e-06 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884646+0000", 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.292e-06 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884648+0000", 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6230000000000001e-06 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892847+0000", 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0081990580000000004 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892891+0000", 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3628999999999999e-05 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892959+0000", 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8151000000000005e-05 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894055+0000", 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001095927 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894088+0000", 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3016999999999999e-05 2024-09-16T04:08:33.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894117+0000", 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9204999999999999e-05 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894140+0000", 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2719000000000001e-05 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.126 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.202363075999997, 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.127 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:33.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:33.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.202309014999997, 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.133 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:33.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:33.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.138 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.978040477, 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:33.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:33.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.400106517000001, 2024-09-16T04:08:33.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:33.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:33.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.146 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s2 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.987332288000001, 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010866819, 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:33.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099684+0000", 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.004e-06 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099714+0000", 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9807000000000001e-05 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099769+0000", 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5306000000000002e-05 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099781+0000", 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2091e-05 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099835+0000", 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4058000000000003e-05 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099845+0000", 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5729999999999997e-06 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110509+0000", 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010664339 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110550+0000", 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640999999999999e-05 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:33.151 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.983293371999999, 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:33.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:33.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.981905957999999, 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:33.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:33.160 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.979638903000001, 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099172860000000009, 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:33.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107377+0000", 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8900000000000001e-07 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107437+0000", 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0628000000000003e-05 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107526+0000", 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8873000000000001e-05 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107535+0000", 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0489999999999993e-06 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111543+0000", 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040080810000000001 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111554+0000", 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0321e-05 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117271+0000", 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057176529999999996 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117294+0000", 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2391999999999998e-05 2024-09-16T04:08:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=188, reqid=client.4440.0:379, at_version=47'56, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.634774501000001, 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011758437, 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452266+0000", 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4151e-05 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459768+0000", 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075025179999999997 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459778+0000", 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0244999999999999e-05 2024-09-16T04:08:33.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459835+0000", 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6394999999999998e-05 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459845+0000", 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0518e-05 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463963+0000", 2024-09-16T04:08:33.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041182450000000004 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463999+0000", 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5667999999999998e-05 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.622586928, 2024-09-16T04:08:33.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:33.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.175 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:426 3.13s0 3:ce079411:::smithi01639990-2:d [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.5795146019999997, 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097940059999999992, 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 426 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.176 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:33.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4799999999999999e-07 2024-09-16T04:08:33.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507505+0000", 2024-09-16T04:08:33.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0149999999999997e-06 2024-09-16T04:08:33.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507511+0000", 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4540000000000004e-06 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507517+0000", 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0749999999999997e-06 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.508922+0000", 2024-09-16T04:08:33.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001405211 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.509012+0000", 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9776000000000002e-05 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.517295+0000", 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082831269999999995 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.64971697799999995, 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019153804, 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437299+0000", 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:08:33.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437320+0000", 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1121999999999999e-05 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437388+0000", 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7889e-05 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437397+0000", 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4390000000000006e-06 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441658+0000", 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042607110000000004 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441667+0000", 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:08:33.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456416+0000", 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014748493 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456452+0000", 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6159999999999999e-05 2024-09-16T04:08:33.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.64652726699999996, 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017174697999999999, 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.186 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440489+0000", 2024-09-16T04:08:33.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.015e-06 2024-09-16T04:08:33.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440509+0000", 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0338000000000001e-05 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440547+0000", 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440557+0000", 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7820000000000005e-06 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442068+0000", 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015112179999999999 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442078+0000", 2024-09-16T04:08:33.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4620000000000002e-06 2024-09-16T04:08:33.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457639+0000", 2024-09-16T04:08:33.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015560954 2024-09-16T04:08:33.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457663+0000", 2024-09-16T04:08:33.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4121000000000002e-05 2024-09-16T04:08:33.192 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.192 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.192 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=82, reqid=osd.8.0:81, at_version=0'0, trim_to=0'0, roll_forward_to=57'51))", 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 0.62680294700000005, 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010694719, 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460213+0000", 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4999999999999999e-07 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460222+0000", 2024-09-16T04:08:33.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2669999999999995e-06 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460651+0000", 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00042901799999999998 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.462020+0000", 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013696680000000001 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:33.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.466451+0000", 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044306529999999997 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.466460+0000", 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.470859+0000", 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043990979999999997 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:33.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:33.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:33.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.470907+0000", 2024-09-16T04:08:33.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7679000000000002e-05 2024-09-16T04:08:33.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.197 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:33.197 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:33.197 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:33.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:33.199 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:33.196+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:33.300 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:33.298+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:33.306 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:33.306 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:33.306 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:33.306 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:33.315 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:33.401 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:33.399+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:33.417 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:33.417 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:33.417 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:33.417 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:33.417 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:33.427 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:33.429 INFO:teuthology.orchestra.run.smithi016.stdout:3.es2 2024-09-16T04:08:33.429 INFO:teuthology.orchestra.run.smithi016.stdout:3.bs6 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.as2 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.5s5 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.4s0 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.19s0 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.6s1 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.1s7 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.18s2 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.17s7 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.7s7 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.16s2 2024-09-16T04:08:33.430 INFO:teuthology.orchestra.run.smithi016.stdout:3.12s3 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:3.13s5 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:3.11s6 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:3.10s6 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:3.9s5 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:3.2s5 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:3.0s3 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:3.8s1 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:2.6 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:2.4 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:3.ds6 2024-09-16T04:08:33.431 INFO:teuthology.orchestra.run.smithi016.stdout:2.2 2024-09-16T04:08:33.437 INFO:tasks.rados.rados.0.smithi016.stdout:197: done (5 left) 2024-09-16T04:08:33.438 INFO:tasks.rados.rados.0.smithi016.stdout:198: done (4 left) 2024-09-16T04:08:33.438 INFO:tasks.rados.rados.0.smithi016.stdout:199: done (3 left) 2024-09-16T04:08:33.438 INFO:tasks.rados.rados.0.smithi016.stdout:200: done (2 left) 2024-09-16T04:08:33.438 INFO:tasks.rados.rados.0.smithi016.stdout:201: done (1 left) 2024-09-16T04:08:33.438 INFO:tasks.rados.rados.0.smithi016.stdout:203: done (0 left) 2024-09-16T04:08:33.438 INFO:tasks.rados.rados.0.smithi016.stdout:204: snap_create 2024-09-16T04:08:33.438 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-16T04:08:33.502 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:33.499+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:33.536 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.429593299, 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073296819999999997, 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103494+0000", 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1399999999999997e-07 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103514+0000", 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9698999999999999e-05 2024-09-16T04:08:33.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103592+0000", 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8378000000000001e-05 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103598+0000", 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9100000000000002e-06 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105790+0000", 2024-09-16T04:08:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021916359999999998 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105798+0000", 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2490000000000001e-06 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110788+0000", 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049902660000000001 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110823+0000", 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5129999999999997e-05 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=34, reqid=client.4440.0:211, at_version=33'20, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.417376362999999, 2024-09-16T04:08:33.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078650630000000003, 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115711+0000", 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2399999999999998e-07 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115771+0000", 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9478999999999998e-05 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119749+0000", 2024-09-16T04:08:33.543 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039784099999999999 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119760+0000", 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0664e-05 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119899+0000", 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013905300000000001 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.544 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119919+0000", 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9854999999999999e-05 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123544+0000", 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036256499999999998 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.545 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123575+0000", 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1328000000000002e-05 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=35, reqid=client.4440.0:212, at_version=33'21, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.414621240999999, 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077855709999999998, 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.547 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118551+0000", 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4878999999999998e-05 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120595+0000", 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020432639999999999 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.548 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120604+0000", 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0250000000000008e-06 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122166+0000", 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015625599999999999 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122175+0000", 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8470000000000002e-06 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.549 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126217+0000", 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040415249999999998 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126251+0000", 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4570999999999998e-05 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:33.550 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:33.551 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.391393478000001, 2024-09-16T04:08:33.551 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:33.551 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.551 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.552 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.552 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.552 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.552 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.552 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:33.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.553 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.553 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:33.553 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.553 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.553 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.554 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:33.554 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:33.554 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.554 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.554 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.554 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:33.554 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:33.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.555 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.555 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.555 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:33.555 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:33.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.555 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.555 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:33.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:33.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.556 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.556 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.556 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:33.556 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:33.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.557 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.557 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.557 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:33.557 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:33.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.558 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:33.558 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:33.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.558 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.558 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.559 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:33.559 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:33.559 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.559 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.559 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.560 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.560 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:296 3.9s0 3:9fd10cc9:::smithi01639990-50:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:08:33.560 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:33.560 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.428513588000001, 2024-09-16T04:08:33.560 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010626313, 2024-09-16T04:08:33.560 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.561 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.561 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.561 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.561 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.561 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.561 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296 2024-09-16T04:08:33.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.562 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.562 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.562 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.562 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:33.562 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.563 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.563 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.563 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:33.563 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.564 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.564 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104575+0000", 2024-09-16T04:08:33.564 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1349999999999999e-06 2024-09-16T04:08:33.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.564 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.564 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.565 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104586+0000", 2024-09-16T04:08:33.565 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0603e-05 2024-09-16T04:08:33.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.565 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.565 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.565 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104606+0000", 2024-09-16T04:08:33.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9840999999999999e-05 2024-09-16T04:08:33.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.566 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.566 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104615+0000", 2024-09-16T04:08:33.566 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0189999999999995e-06 2024-09-16T04:08:33.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.567 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.567 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.567 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104674+0000", 2024-09-16T04:08:33.567 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9534999999999998e-05 2024-09-16T04:08:33.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.568 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.568 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:33.568 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104680+0000", 2024-09-16T04:08:33.568 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9320000000000001e-06 2024-09-16T04:08:33.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.569 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.569 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109134+0000", 2024-09-16T04:08:33.569 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044534120000000003 2024-09-16T04:08:33.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.569 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.570 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.570 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109195+0000", 2024-09-16T04:08:33.570 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0983000000000002e-05 2024-09-16T04:08:33.570 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.570 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.570 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109711+0000", 2024-09-16T04:08:33.571 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051621600000000001 2024-09-16T04:08:33.571 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.571 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.571 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.571 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.110753+0000", 2024-09-16T04:08:33.572 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001042533 2024-09-16T04:08:33.572 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.572 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.572 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:33.572 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115115+0000", 2024-09-16T04:08:33.573 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043613030000000004 2024-09-16T04:08:33.573 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.573 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.573 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.573 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115200+0000", 2024-09-16T04:08:33.574 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4801000000000003e-05 2024-09-16T04:08:33.574 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.574 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.574 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.574 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.574 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.575 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:33.575 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:33.575 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.084064217000002, 2024-09-16T04:08:33.575 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085197069999999996, 2024-09-16T04:08:33.575 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4000000000000001e-07 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449160+0000", 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013637999999999999 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.577 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451237+0000", 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020768179999999998 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451246+0000", 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451532+0000", 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00028571800000000002 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.578 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451542+0000", 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0118999999999999e-05 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457500+0000", 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059576680000000002 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457542+0000", 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2716999999999998e-05 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.579 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.07333787, 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.580 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:33.581 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.582 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.582 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.582 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:08:33.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:08:33.582 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.582 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.582 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.582 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:08:33.582 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.583 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:08:33.584 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.942795711, 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:363 3.9s0 3:94de6fe1:::smithi01639990-3:head [stat,delete] snapc 8=[6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.093229051, 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085012569999999999, 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 363 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439860+0000", 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439869+0000", 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3170000000000008e-06 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439888+0000", 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9293e-05 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439897+0000", 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439953+0000", 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6317999999999998e-05 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439959+0000", 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3040000000000004e-06 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441596+0000", 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001637655 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441659+0000", 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2658999999999998e-05 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.442297+0000", 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063811800000000002 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.443562+0000", 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001265361 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448304+0000", 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047413560000000004 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448359+0000", 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5189000000000003e-05 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.861873158, 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.701469100000001, 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:419 3.9s0 3:94de6fe1:::smithi01639990-3:head [append 0~786432] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.0305750260000002, 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082147929999999997, 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 419 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502514+0000", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.65e-06 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503628+0000", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001113239 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503646+0000", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7906000000000001e-05 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503653+0000", 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2339999999999997e-06 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503741+0000", 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8436999999999994e-05 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503820+0000", 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8751000000000005e-05 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505036+0000", 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001216017 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510097+0000", 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050614550000000003 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510625+0000", 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000527667 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510727+0000", 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101437 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:420 3.9s0 3:94de6fe1:::smithi01639990-3:head [append 0~491520] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.0279431250000002, 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089351980000000001, 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 420 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505614+0000", 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00046960099999999999 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505627+0000", 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3275000000000001e-05 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505636+0000", 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2609999999999993e-06 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505739+0000", 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010322700000000001 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505847+0000", 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107929 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.506958+0000", 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011114580000000001 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510232+0000", 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032740080000000001 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.513941+0000", 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037089470000000002 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514079+0000", 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000137744 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:422 3.9s0 3:94de6fe1:::smithi01639990-3:head [setxattr _header (56) in=7b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507590+0000", 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.0254968900000003, 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007312804, 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 422 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507590+0000", 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507590+0000", 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507591+0000", 2024-09-16T04:08:33.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507594+0000", 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5329999999999999e-06 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507602+0000", 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5379999999999997e-06 2024-09-16T04:08:33.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507607+0000", 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4500000000000003e-06 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.509353+0000", 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001745468 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.509454+0000", 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101718 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.509859+0000", 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040448799999999999 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.513811+0000", 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039521690000000002 2024-09-16T04:08:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514806+0000", 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099465199999999995 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514903+0000", 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6979999999999996e-05 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:423 3.9s0 3:94de6fe1:::smithi01639990-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.0254707490000001, 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010412819, 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 423 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507617+0000", 2024-09-16T04:08:33.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6599999999999996e-07 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507621+0000", 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9609999999999999e-06 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507628+0000", 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3139999999999998e-06 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507637+0000", 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1840000000000005e-06 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510428+0000", 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027909060000000001 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510505+0000", 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6679000000000002e-05 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514954+0000", 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044494410000000002 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515004+0000", 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0244999999999998e-05 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518029+0000", 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030245229999999999 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.633 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:427 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507644+0000", 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.0254424350000004, 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079136850000000002, 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 427 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507644+0000", 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507644+0000", 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507645+0000", 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4499999999999997e-07 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507649+0000", 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5789999999999997e-06 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507655+0000", 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6729999999999997e-06 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507660+0000", 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0329999999999999e-06 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510813+0000", 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031532259999999999 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510938+0000", 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000124757 2024-09-16T04:08:33.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.511503+0000", 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00056476499999999995 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514248+0000", 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002745272 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515498+0000", 2024-09-16T04:08:33.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012494139999999999 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515558+0000", 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0520999999999999e-05 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:428 3.9s0 3:92dcccbe:::smithi01639990-16:head [rollback c] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.0254083639999996, 2024-09-16T04:08:33.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0081187840000000004, 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 428 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507679+0000", 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0900000000000002e-07 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507684+0000", 2024-09-16T04:08:33.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3199999999999999e-06 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507691+0000", 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1160000000000001e-06 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507697+0000", 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3129999999999998e-06 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512167+0000", 2024-09-16T04:08:33.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044708880000000001 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512254+0000", 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6329999999999995e-05 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512662+0000", 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040792999999999998 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514513+0000", 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018516019999999999 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515756+0000", 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001242976 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515797+0000", 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0899999999999998e-05 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.644 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.0253642630000002, 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.645 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:08:33.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:08:33.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0036512379999998, 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.651 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:08:33.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.0035990110000004, 2024-09-16T04:08:33.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:08:33.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:33.661 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:33.663 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:33.664 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:33.661+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:33.764 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:33.762+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:33.807 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:33.807 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:33.807 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:33.807 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:33.817 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:33.866 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:33.863+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:33.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:33.919 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:33.920 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:33.920 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:33.920 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:33.920 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:33.928 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:33.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 3.es2 --file /var/log/ceph/exp.3.es2.1 2024-09-16T04:08:33.967 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:33.966+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:34.059 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:34.059 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:34.059 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T04:08:34.069 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:34.067+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:34.074 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:34.074 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:34.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:34.074 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:34.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.074 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:34.075 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:34.075 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.977050699999999, 2024-09-16T04:08:34.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:34.075 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.075 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.075 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.075 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:34.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:34.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:34.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:34.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:34.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:34.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.971610226999999, 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067064200000000003, 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.080 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099749+0000", 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1299999999999999e-07 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099771+0000", 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2642000000000001e-05 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099811+0000", 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9406999999999997e-05 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099819+0000", 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100401+0000", 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058156199999999999 2024-09-16T04:08:34.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100411+0000", 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7079999999999992e-06 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106408+0000", 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059972369999999999 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106454+0000", 2024-09-16T04:08:34.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6384000000000002e-05 2024-09-16T04:08:34.085 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.085 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.085 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.085 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:34.085 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:34.085 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.967163315000001, 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062253660000000004, 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:34.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104196+0000", 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104218+0000", 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1974999999999999e-05 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104245+0000", 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7489999999999999e-05 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104254+0000", 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104313+0000", 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9151000000000003e-05 2024-09-16T04:08:34.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104322+0000", 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110389+0000", 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006067205 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110420+0000", 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1100999999999999e-05 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.961346793000001, 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062070839999999999, 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.090 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110012+0000", 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3999999999999997e-07 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110025+0000", 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3053e-05 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110784+0000", 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075944100000000002 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110793+0000", 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4880000000000007e-06 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110853+0000", 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0333e-05 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110861+0000", 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2090000000000005e-06 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116184+0000", 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053227389999999999 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116218+0000", 2024-09-16T04:08:34.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4480999999999997e-05 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 37/27 ECSubWrite(tid=129, reqid=osd.3.0:129, at_version=37'43, trim_to=0'0, roll_forward_to=35'41))", 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.958581006999999, 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061976449999999999, 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112778+0000", 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3499999999999996e-07 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112797+0000", 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9578000000000001e-05 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112842+0000", 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5093999999999999e-05 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112850+0000", 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112972+0000", 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000122033 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112982+0000", 2024-09-16T04:08:34.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5789999999999993e-06 2024-09-16T04:08:34.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118944+0000", 2024-09-16T04:08:34.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005961816 2024-09-16T04:08:34.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118975+0000", 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0933999999999999e-05 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=155, reqid=osd.3.0:155, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.631717894000001, 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073085160000000001, 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.100 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1299999999999999e-07 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439663+0000", 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3030000000000001e-05 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439705+0000", 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1078999999999998e-05 2024-09-16T04:08:34.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439714+0000", 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441461+0000", 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017466350000000001 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441475+0000", 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4404e-05 2024-09-16T04:08:34.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446911+0000", 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054361430000000001 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446949+0000", 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7465000000000001e-05 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.627639126999998, 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068399539999999997, 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:34.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:34.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443720+0000", 2024-09-16T04:08:34.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9500000000000001e-07 2024-09-16T04:08:34.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444146+0000", 2024-09-16T04:08:34.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042652899999999998 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444178+0000", 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1412999999999999e-05 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444185+0000", 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4270000000000002e-06 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444247+0000", 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1760000000000005e-05 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444254+0000", 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9260000000000003e-06 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450519+0000", 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062649960000000001 2024-09-16T04:08:34.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450559+0000", 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0108000000000003e-05 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s5 42/27 ECSubWrite(tid=40, reqid=osd.0.0:40, at_version=42'11, trim_to=0'0, roll_forward_to=41'9))", 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.627465119, 2024-09-16T04:08:34.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067397389999999998, 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:34.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0700000000000001e-07 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443914+0000", 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1274e-05 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443937+0000", 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2991999999999999e-05 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443945+0000", 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4780000000000002e-06 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444021+0000", 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5814000000000003e-05 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444033+0000", 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2322e-05 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450586+0000", 2024-09-16T04:08:34.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065529869999999997 2024-09-16T04:08:34.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450633+0000", 2024-09-16T04:08:34.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6764999999999997e-05 2024-09-16T04:08:34.116 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.116 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.641533730000001, 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.117 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:08:34.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.181583503000001, 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.121 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:08:34.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:08:34.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:08:34.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.858070897999999, 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:34.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:08:34.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:08:34.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:08:34.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:08:34.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:08:34.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:08:34.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:08:34.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:08:34.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:34.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:08:34.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:08:34.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:08:34.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:08:34.132 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.133 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.133 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.133 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:34.133 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:34.133 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.60028704, 2024-09-16T04:08:34.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078488229999999996, 2024-09-16T04:08:34.134 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.134 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.134 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.134 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:34.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:34.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471072+0000", 2024-09-16T04:08:34.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0279999999999999e-06 2024-09-16T04:08:34.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471418+0000", 2024-09-16T04:08:34.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034541800000000001 2024-09-16T04:08:34.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471457+0000", 2024-09-16T04:08:34.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:34.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471492+0000", 2024-09-16T04:08:34.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:08:34.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471798+0000", 2024-09-16T04:08:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030615599999999997 2024-09-16T04:08:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471812+0000", 2024-09-16T04:08:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.448e-05 2024-09-16T04:08:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478881+0000", 2024-09-16T04:08:34.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070685549999999998 2024-09-16T04:08:34.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478920+0000", 2024-09-16T04:08:34.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9072000000000001e-05 2024-09-16T04:08:34.140 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.141 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.141 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.141 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 53/27 ECSubWrite(tid=103, reqid=osd.5.0:103, at_version=53'29, trim_to=0'0, roll_forward_to=52'27))", 2024-09-16T04:08:34.141 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:34.141 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.5837303819999997, 2024-09-16T04:08:34.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068898910000000004, 2024-09-16T04:08:34.141 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.142 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.142 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.142 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:34.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:34.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:34.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.22e-07 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487655+0000", 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.669e-05 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487696+0000", 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0930000000000003e-05 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487708+0000", 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2699e-05 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487808+0000", 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9557999999999998e-05 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487819+0000", 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1297999999999999e-05 2024-09-16T04:08:34.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.494471+0000", 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0066521230000000002 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.494518+0000", 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6371e-05 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.147 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 53/46 ECSubWrite(tid=213, reqid=osd.3.0:213, at_version=53'65, trim_to=0'0, roll_forward_to=52'63))", 2024-09-16T04:08:34.147 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:34.147 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.5809593690000003, 2024-09-16T04:08:34.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070799060000000004, 2024-09-16T04:08:34.147 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.147 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.147 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.148 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:34.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:34.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490424+0000", 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5196999999999999e-05 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490482+0000", 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7442000000000003e-05 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490499+0000", 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7467999999999999e-05 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490695+0000", 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000195897 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490707+0000", 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2221999999999999e-05 2024-09-16T04:08:34.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.497426+0000", 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067186169999999996 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.497479+0000", 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2876e-05 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.152 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 53/46 ECSubWrite(tid=216, reqid=osd.3.0:213, at_version=0'0, trim_to=0'0, roll_forward_to=53'65))", 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.498161+0000", 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.573197253, 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059890799999999999, 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.153 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498161+0000", 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498161+0000", 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498162+0000", 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9299999999999996e-07 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498174+0000", 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2391e-05 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498188+0000", 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3835000000000001e-05 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498194+0000", 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9869999999999996e-06 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498264+0000", 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0053000000000002e-05 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498276+0000", 2024-09-16T04:08:34.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2251e-05 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.504086+0000", 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005809982 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.504150+0000", 2024-09-16T04:08:34.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3688000000000004e-05 2024-09-16T04:08:34.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.158 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.158 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.15s7 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:34.159 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.438296+0000", 2024-09-16T04:08:34.159 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.6330618430000001, 2024-09-16T04:08:34.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064265700000000004, 2024-09-16T04:08:34.159 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.159 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.159 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.160 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438296+0000", 2024-09-16T04:08:34.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438296+0000", 2024-09-16T04:08:34.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438297+0000", 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.47e-07 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438328+0000", 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1115000000000003e-05 2024-09-16T04:08:34.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438376+0000", 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7774e-05 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438385+0000", 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6719999999999993e-06 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438449+0000", 2024-09-16T04:08:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4206000000000002e-05 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438460+0000", 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0862e-05 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.444681+0000", 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062210820000000002 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.444723+0000", 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2011999999999997e-05 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.6311725720000001, 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084259260000000002, 2024-09-16T04:08:34.165 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.166 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.166 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.166 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:34.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:34.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.91e-07 2024-09-16T04:08:34.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440211+0000", 2024-09-16T04:08:34.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4978999999999999e-05 2024-09-16T04:08:34.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440257+0000", 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6569000000000001e-05 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440268+0000", 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1056000000000001e-05 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440348+0000", 2024-09-16T04:08:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9795000000000004e-05 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440364+0000", 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5996999999999999e-05 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448570+0000", 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082059590000000005 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448611+0000", 2024-09-16T04:08:34.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.138e-05 2024-09-16T04:08:34.171 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.171 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.171 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.bs7 57/56 [PushOp(3:d683cb60:::smithi01639990-49:head, version: 53'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d683cb60:::smithi01639990-49:head@53'38, size: 2310144, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.510590401, 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.043702234999999999, 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:34.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560769+0000", 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0049999999999999e-06 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.561301+0000", 2024-09-16T04:08:34.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053276500000000004 2024-09-16T04:08:34.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603955+0000", 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042653605999999997 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603967+0000", 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2065000000000001e-05 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604063+0000", 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5599000000000004e-05 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604470+0000", 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040719500000000001 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=252, reqid=osd.3.0:252, at_version=57'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.1618642610000001, 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27497091299999998, 2024-09-16T04:08:34.177 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.178 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.178 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.178 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:34.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:34.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:34.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:08:34.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909517+0000", 2024-09-16T04:08:34.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2886000000000001e-05 2024-09-16T04:08:34.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183847+0000", 2024-09-16T04:08:34.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27432981400000001 2024-09-16T04:08:34.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183853+0000", 2024-09-16T04:08:34.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2929999999999997e-06 2024-09-16T04:08:34.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183902+0000", 2024-09-16T04:08:34.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9008999999999998e-05 2024-09-16T04:08:34.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183911+0000", 2024-09-16T04:08:34.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:08:34.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184435+0000", 2024-09-16T04:08:34.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052493699999999995 2024-09-16T04:08:34.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184465+0000", 2024-09-16T04:08:34.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9292e-05 2024-09-16T04:08:34.186 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.186 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.186 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.188 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=253, reqid=osd.3.0:253, at_version=57'67, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:34.188 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:34.188 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.1612362300000001, 2024-09-16T04:08:34.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50851210899999999, 2024-09-16T04:08:34.188 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:34.188 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:34.188 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:34.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910123+0000", 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910136+0000", 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.331e-05 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416910+0000", 2024-09-16T04:08:34.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50677369000000005 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416919+0000", 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8290000000000003e-06 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416975+0000", 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5983999999999998e-05 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416985+0000", 2024-09-16T04:08:34.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418599+0000", 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001614515 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418634+0000", 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4588999999999997e-05 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.192 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.193 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:34.193 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:34.193 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:34.194 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:34.194 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:34.195 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:34.195 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:34.195 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:34.196 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:34.196 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:34.196 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:34.197 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:34.197 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:34.197 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:34.197 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:34.198 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:34.198 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:34.192+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:34.212 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:34.234 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:34.234 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:34.255 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:34.255 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:34.276 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:34.277 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:08:34.280 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:08:34.295 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:34.292+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:34.303 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:34.303 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:34.312 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:34.396 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:34.394+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:34.438 INFO:tasks.rados.rados.0.smithi016.stdout:204: done (0 left) 2024-09-16T04:08:34.438 INFO:tasks.rados.rados.0.smithi016.stdout:205: snap_remove snap 12 2024-09-16T04:08:34.458 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:34.458 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:34.467 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:34.497 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:34.494+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:34.570 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:34.570 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:34.579 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:34.598 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:34.596+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:34.682 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:34.682 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:34.691 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:34.699 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:34.696+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:34.794 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:34.795 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:34.800 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:34.798+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:34.803 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:34.901 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:34.899+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:34.906 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:34.906 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:34.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:35.002 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:35.000+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:35.025 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:35.026 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:35.034 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:35.103 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:35.100+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:35.185 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:35.185 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:35.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:35.209 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:35.207+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:35.304 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:35.304 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:35.305 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:35.303+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:35.313 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:35.405 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:35.404+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:35.439 INFO:tasks.rados.rados.0.smithi016.stdout:205: done (0 left) 2024-09-16T04:08:35.439 INFO:tasks.rados.rados.0.smithi016.stdout:206: read oid 47 snap -1 2024-09-16T04:08:35.439 INFO:tasks.rados.rados.0.smithi016.stdout:206: expect (ObjNum 94 snap 12 seq_num 94) 2024-09-16T04:08:35.440 INFO:tasks.rados.rados.0.smithi016.stdout:207: delete oid 22 current snap is 17 2024-09-16T04:08:35.451 INFO:tasks.rados.rados.0.smithi016.stdout:207: done (1 left) 2024-09-16T04:08:35.451 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 8 current snap is 17 2024-09-16T04:08:35.451 INFO:tasks.rados.rados.0.smithi016.stdout:208: seq_num 100 ranges {131072=524288,655360=753664,1409024=409600} 2024-09-16T04:08:35.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:08:35.453 INFO:tasks.rados.rados.0.smithi016.stdout:208: writing smithi01639990-8 from 131072 to 655360 tid 1 2024-09-16T04:08:35.460 INFO:tasks.rados.rados.0.smithi016.stdout:208: writing smithi01639990-8 from 655360 to 1409024 tid 2 2024-09-16T04:08:35.464 INFO:tasks.rados.rados.0.smithi016.stdout:208: writing smithi01639990-8 from 1409024 to 1818624 tid 3 2024-09-16T04:08:35.464 INFO:tasks.rados.rados.0.smithi016.stdout:209: rollback oid 23 current snap is 17 2024-09-16T04:08:35.464 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 23 to 15 2024-09-16T04:08:35.464 INFO:tasks.rados.rados.0.smithi016.stdout:208: finishing write tid 1 to smithi01639990-8 2024-09-16T04:08:35.465 INFO:tasks.rados.rados.0.smithi016.stdout:210: snap_remove snap 9 2024-09-16T04:08:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:35.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:35.506 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:35.504+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:35.607 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:35.604+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:35.636 INFO:teuthology.orchestra.run.smithi016.stderr:Exporting 3.es2 info 3.es2( v 34'8 (0'0,34'8] local-lis/les=27/28 n=1 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi016.stderr:Read 2#3:7f241f48:::smithi01639990-30:head# 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi016.stderr:Export successful 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:35.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:35.713 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:35.710+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:35.770 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:35.771 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:35.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:35.811 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:35.809+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:35.887 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:35.887 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:35.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:35.913 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:35.910+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:36.005 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:36.005 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:36.014 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:36.014 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:36.011+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:36.114 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:36.112+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:36.121 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:36.121 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:36.130 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:36.171 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 3.es2 2024-09-16T04:08:36.214 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:36.212+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:36.247 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:36.247 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:36.256 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:36.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:36.327 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:36.325+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:36.362 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:36.362 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:36.370 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:36.416 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:36.414+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:36.418 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T04:08:36.438 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-16T04:08:36.441 INFO:tasks.rados.rados.0.smithi016.stdout:210: done (3 left) 2024-09-16T04:08:36.441 INFO:tasks.rados.rados.0.smithi016.stdout:211: snap_remove snap 13 2024-09-16T04:08:36.442 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:36.442 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:36.471 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:36.488 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:36.488 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:36.497 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:36.545 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:36.542+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:36.603 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:36.603 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:36.612 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:36.618 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:36.616+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:36.718 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:36.715+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:36.755 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:36.756 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:36.764 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:36.818 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:36.816+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:36.870 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:36.871 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:36.879 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:36.920 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:36.918+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:37.021 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:37.018+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:37.023 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:37.023 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:37.032 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:37.122 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:37.120+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:37.134 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:37.134 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:37.143 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:37.222 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:37.220+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:37.247 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:37.247 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:37.255 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:37.297 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:37.297 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:37.318 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:37.319 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:37.323 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:37.321+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:37.343 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:37.343 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:37.366 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:37.366 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:37.388 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:37.400 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:37.400 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:37.400 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:37.400 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:37.408 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:37.410 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:37.410 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:37.432 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:37.433 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:37.442 INFO:tasks.rados.rados.0.smithi016.stdout:211: done (3 left) 2024-09-16T04:08:37.442 INFO:tasks.rados.rados.0.smithi016.stdout:212: setattr oid 26 current snap is 17 2024-09-16T04:08:37.442 INFO:tasks.rados.rados.0.smithi016.stdout:206: expect (ObjNum 94 snap 12 seq_num 94) 2024-09-16T04:08:37.453 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:37.450+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:37.461 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:37.461 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:08:37.466 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:08:37.523 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:37.523 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:37.523 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:37.523 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:37.523 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:37.524 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:37.522+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:37.532 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:208: finishing write tid 2 to smithi01639990-8 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:208: finishing write tid 3 to smithi01639990-8 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:209: finishing rollback tid 1 to smithi01639990-23 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 23 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:208: finishing write tid 4 to smithi01639990-8 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:208: finishing write tid 5 to smithi01639990-8 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:208: oid 8 updating version 0 to 28 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:208: oid 8 version 28 is already newer than 27 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:208: oid 8 updating version 28 to 29 2024-09-16T04:08:37.577 INFO:tasks.rados.rados.0.smithi016.stdout:208: oid 8 version 29 is already newer than 26 2024-09-16T04:08:37.578 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 8 v 29 (ObjNum 100 snap 17 seq_num 100) dirty exists 2024-09-16T04:08:37.578 INFO:tasks.rados.rados.0.smithi016.stdout:208: left oid 8 (ObjNum 100 snap 17 seq_num 100) 2024-09-16T04:08:37.578 INFO:tasks.rados.rados.0.smithi016.stdout:206: done (3 left) 2024-09-16T04:08:37.578 INFO:tasks.rados.rados.0.smithi016.stdout:208: done (2 left) 2024-09-16T04:08:37.578 INFO:tasks.rados.rados.0.smithi016.stdout:209: done (1 left) 2024-09-16T04:08:37.578 INFO:tasks.rados.rados.0.smithi016.stdout:213: rmattr oid 46 current snap is 17 2024-09-16T04:08:37.578 INFO:tasks.rados.rados.0.smithi016.stdout:213: done (1 left) 2024-09-16T04:08:37.578 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 9 current snap is 17 2024-09-16T04:08:37.578 INFO:tasks.rados.rados.0.smithi016.stdout:214: seq_num 101 ranges {147456=540672,688128=507904,1196032=622592,1818624=32768} 2024-09-16T04:08:37.580 INFO:tasks.rados.rados.0.smithi016.stdout:214: writing smithi01639990-9 from 147456 to 688128 tid 1 2024-09-16T04:08:37.584 INFO:tasks.rados.rados.0.smithi016.stdout:214: writing smithi01639990-9 from 688128 to 1196032 tid 2 2024-09-16T04:08:37.588 INFO:tasks.rados.rados.0.smithi016.stdout:214: writing smithi01639990-9 from 1196032 to 1818624 tid 3 2024-09-16T04:08:37.588 INFO:tasks.rados.rados.0.smithi016.stdout:214: writing smithi01639990-9 from 1818624 to 1851392 tid 4 2024-09-16T04:08:37.588 INFO:tasks.rados.rados.0.smithi016.stdout:215: delete oid 36 current snap is 17 2024-09-16T04:08:37.588 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 26 v 17 (ObjNum 80 snap 7 seq_num 80) dirty exists 2024-09-16T04:08:37.588 INFO:tasks.rados.rados.0.smithi016.stdout:214: finishing write tid 1 to smithi01639990-9 2024-09-16T04:08:37.596 INFO:tasks.rados.rados.0.smithi016.stdout:214: finishing write tid 2 to smithi01639990-9 2024-09-16T04:08:37.596 INFO:tasks.rados.rados.0.smithi016.stdout:214: finishing write tid 3 to smithi01639990-9 2024-09-16T04:08:37.597 INFO:tasks.rados.rados.0.smithi016.stdout:214: finishing write tid 4 to smithi01639990-9 2024-09-16T04:08:37.597 INFO:tasks.rados.rados.0.smithi016.stdout:214: finishing write tid 5 to smithi01639990-9 2024-09-16T04:08:37.598 INFO:tasks.rados.rados.0.smithi016.stdout:212: done (2 left) 2024-09-16T04:08:37.598 INFO:tasks.rados.rados.0.smithi016.stdout:215: done (1 left) 2024-09-16T04:08:37.598 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 30 current snap is 17 2024-09-16T04:08:37.598 INFO:tasks.rados.rados.0.smithi016.stdout:216: seq_num 103 ranges {2080768=573440,2654208=786432,3440640=376832} 2024-09-16T04:08:37.613 INFO:tasks.rados.rados.0.smithi016.stdout:216: writing smithi01639990-30 from 2080768 to 2654208 tid 1 2024-09-16T04:08:37.617 INFO:tasks.rados.rados.0.smithi016.stdout:216: writing smithi01639990-30 from 2654208 to 3440640 tid 2 2024-09-16T04:08:37.619 INFO:teuthology.orchestra.run.smithi016.stdout: marking collection for removal 2024-09-16T04:08:37.619 INFO:teuthology.orchestra.run.smithi016.stdout:setting '_remove' omap key 2024-09-16T04:08:37.619 INFO:teuthology.orchestra.run.smithi016.stdout:finish_remove_pgs 3.es2_head removing 3.es2 2024-09-16T04:08:37.620 INFO:tasks.rados.rados.0.smithi016.stdout:216: writing smithi01639990-30 from 3440640 to 3817472 tid 3 2024-09-16T04:08:37.620 INFO:tasks.rados.rados.0.smithi016.stdout:217: rmattr oid 35 current snap is 17 2024-09-16T04:08:37.620 INFO:tasks.rados.rados.0.smithi016.stdout:216: finishing write tid 1 to smithi01639990-30 2024-09-16T04:08:37.621 INFO:tasks.rados.rados.0.smithi016.stdout:218: copy_from oid 14 from oid 47 current snap is 17 2024-09-16T04:08:37.621 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 23 current snap is 17 2024-09-16T04:08:37.621 INFO:tasks.rados.rados.0.smithi016.stdout:219: seq_num 104 ranges {0=589824,589824=540672,1130496=622592} 2024-09-16T04:08:37.621 INFO:teuthology.orchestra.run.smithi016.stdout:Remove successful 2024-09-16T04:08:37.623 INFO:tasks.rados.rados.0.smithi016.stdout:219: writing smithi01639990-23 from 0 to 589824 tid 1 2024-09-16T04:08:37.624 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:37.623+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:37.627 INFO:tasks.rados.rados.0.smithi016.stdout:219: writing smithi01639990-23 from 589824 to 1130496 tid 2 2024-09-16T04:08:37.631 INFO:tasks.rados.rados.0.smithi016.stdout:219: writing smithi01639990-23 from 1130496 to 1753088 tid 3 2024-09-16T04:08:37.631 INFO:tasks.rados.rados.0.smithi016.stdout:220: rmattr oid 43 current snap is 17 2024-09-16T04:08:37.631 INFO:tasks.rados.rados.0.smithi016.stdout:220: done (5 left) 2024-09-16T04:08:37.631 INFO:tasks.rados.rados.0.smithi016.stdout:221: delete oid 28 current snap is 17 2024-09-16T04:08:37.631 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 35 v 0 (ObjNum 69 snap 3 seq_num 69) dirty exists 2024-09-16T04:08:37.632 INFO:tasks.rados.rados.0.smithi016.stdout:216: finishing write tid 2 to smithi01639990-30 2024-09-16T04:08:37.632 INFO:tasks.rados.rados.0.smithi016.stdout:216: finishing write tid 3 to smithi01639990-30 2024-09-16T04:08:37.632 INFO:tasks.rados.rados.0.smithi016.stdout:216: finishing write tid 4 to smithi01639990-30 2024-09-16T04:08:37.632 INFO:tasks.rados.rados.0.smithi016.stdout:219: finishing write tid 1 to smithi01639990-23 2024-09-16T04:08:37.632 INFO:tasks.rados.rados.0.smithi016.stdout:214: finishing write tid 6 to smithi01639990-9 2024-09-16T04:08:37.632 INFO:tasks.rados.rados.0.smithi016.stdout:214: oid 9 updating version 0 to 46 2024-09-16T04:08:37.632 INFO:tasks.rados.rados.0.smithi016.stdout:214: oid 9 version 46 is already newer than 44 2024-09-16T04:08:37.632 INFO:tasks.rados.rados.0.smithi016.stdout:214: oid 9 version 46 is already newer than 45 2024-09-16T04:08:37.633 INFO:tasks.rados.rados.0.smithi016.stdout:214: oid 9 version 46 is already newer than 42 2024-09-16T04:08:37.633 INFO:tasks.rados.rados.0.smithi016.stdout:214: oid 9 version 46 is already newer than 43 2024-09-16T04:08:37.633 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 9 v 46 (ObjNum 101 snap 17 seq_num 101) dirty exists 2024-09-16T04:08:37.633 INFO:tasks.rados.rados.0.smithi016.stdout:214: left oid 9 (ObjNum 101 snap 17 seq_num 101) 2024-09-16T04:08:37.635 INFO:tasks.rados.rados.0.smithi016.stdout:219: finishing write tid 2 to smithi01639990-23 2024-09-16T04:08:37.637 INFO:tasks.rados.rados.0.smithi016.stdout:216: finishing write tid 5 to smithi01639990-30 2024-09-16T04:08:37.637 INFO:tasks.rados.rados.0.smithi016.stdout:216: oid 30 updating version 0 to 27 2024-09-16T04:08:37.637 INFO:tasks.rados.rados.0.smithi016.stdout:216: oid 30 version 27 is already newer than 24 2024-09-16T04:08:37.637 INFO:tasks.rados.rados.0.smithi016.stdout:216: oid 30 version 27 is already newer than 25 2024-09-16T04:08:37.637 INFO:tasks.rados.rados.0.smithi016.stdout:216: oid 30 version 27 is already newer than 26 2024-09-16T04:08:37.637 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 30 v 27 (ObjNum 103 snap 17 seq_num 103) dirty exists 2024-09-16T04:08:37.637 INFO:tasks.rados.rados.0.smithi016.stdout:216: left oid 30 (ObjNum 103 snap 17 seq_num 103) 2024-09-16T04:08:37.646 INFO:tasks.rados.rados.0.smithi016.stdout:219: finishing write tid 3 to smithi01639990-23 2024-09-16T04:08:37.647 INFO:tasks.rados.rados.0.smithi016.stdout:218: finishing copy_from racing read to smithi01639990-14 2024-09-16T04:08:37.647 INFO:tasks.rados.rados.0.smithi016.stdout:219: finishing write tid 4 to smithi01639990-23 2024-09-16T04:08:37.686 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:37.686 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:37.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:37.687 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:37.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.687 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:37.687 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:37.687 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 47.742733842, 2024-09-16T04:08:37.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:37.687 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.688 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.688 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:37.689 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:37.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:37.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:37.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 46.760329599000002, 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.696 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.697 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:37.697 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:37.697 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:37.697 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:37.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:37.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:37.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:37.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:37.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:37.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:37.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:37.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:37.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:37.707 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.707 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.707 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.707 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:37.707 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 46.749967093000002, 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:37.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:37.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.253379041000002, 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:37.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:37.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:37.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:37.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:37.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:37.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:37.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:37.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:37.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:37.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:37.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:37.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:37.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:37.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:37.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:37.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:37.719 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.719 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.720 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.720 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:162 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:37.720 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:37.720 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.798830457000001, 2024-09-16T04:08:37.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094233230000000008, 2024-09-16T04:08:37.720 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.721 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.721 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:37.721 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:37.721 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:37.721 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:37.721 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 162 2024-09-16T04:08:37.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.722 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:37.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884626+0000", 2024-09-16T04:08:37.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884627+0000", 2024-09-16T04:08:37.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:08:37.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884629+0000", 2024-09-16T04:08:37.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.187e-06 2024-09-16T04:08:37.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884632+0000", 2024-09-16T04:08:37.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.501e-06 2024-09-16T04:08:37.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884635+0000", 2024-09-16T04:08:37.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7389999999999999e-06 2024-09-16T04:08:37.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892223+0000", 2024-09-16T04:08:37.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075881029999999997 2024-09-16T04:08:37.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:37.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892291+0000", 2024-09-16T04:08:37.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7508000000000005e-05 2024-09-16T04:08:37.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:37.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892412+0000", 2024-09-16T04:08:37.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012124 2024-09-16T04:08:37.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:37.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893979+0000", 2024-09-16T04:08:37.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015676069999999999 2024-09-16T04:08:37.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:37.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893987+0000", 2024-09-16T04:08:37.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.306e-06 2024-09-16T04:08:37.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:37.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894013+0000", 2024-09-16T04:08:37.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6305999999999999e-05 2024-09-16T04:08:37.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894050+0000", 2024-09-16T04:08:37.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6492999999999998e-05 2024-09-16T04:08:37.730 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.731 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.731 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.731 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:163 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:37.731 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:37.731 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.798818085999997, 2024-09-16T04:08:37.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095013200000000006, 2024-09-16T04:08:37.732 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.732 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.732 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:37.732 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:37.732 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:37.732 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:37.732 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 163 2024-09-16T04:08:37.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.733 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884639+0000", 2024-09-16T04:08:37.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2100000000000001e-07 2024-09-16T04:08:37.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884642+0000", 2024-09-16T04:08:37.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.478e-06 2024-09-16T04:08:37.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884646+0000", 2024-09-16T04:08:37.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.292e-06 2024-09-16T04:08:37.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884648+0000", 2024-09-16T04:08:37.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6230000000000001e-06 2024-09-16T04:08:37.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892847+0000", 2024-09-16T04:08:37.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0081990580000000004 2024-09-16T04:08:37.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892891+0000", 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3628999999999999e-05 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.892959+0000", 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8151000000000005e-05 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894055+0000", 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001095927 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894088+0000", 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3016999999999999e-05 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894117+0000", 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9204999999999999e-05 2024-09-16T04:08:37.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894140+0000", 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2719000000000001e-05 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.740 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.798804548, 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:37.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:37.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:37.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:37.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:37.747 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.798750487, 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.748 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:37.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:37.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:37.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:37.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:37.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:37.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:37.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:37.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:37.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:37.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:37.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:37.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:37.756 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.756 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.756 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.756 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:37.757 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:37.757 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.574481948999999, 2024-09-16T04:08:37.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:37.757 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.757 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.757 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.757 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:37.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:37.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:37.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:37.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:37.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:37.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:37.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:37.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:37.762 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:37.762 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.762 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.762 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:37.762 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:37.762 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.762 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.763 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:37.763 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:37.763 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.763 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.763 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.764 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.764 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:37.764 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:37.764 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.996547989, 2024-09-16T04:08:37.765 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:37.765 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.765 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.765 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.765 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.765 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:37.766 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:37.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.767 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.767 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.767 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:37.767 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:37.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.767 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.767 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:37.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.769 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s2 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.58377376, 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010866819, 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.770 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099684+0000", 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.004e-06 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099714+0000", 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9807000000000001e-05 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099769+0000", 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5306000000000002e-05 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.772 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.773 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.773 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099781+0000", 2024-09-16T04:08:37.773 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2091e-05 2024-09-16T04:08:37.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.773 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.773 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.773 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099835+0000", 2024-09-16T04:08:37.773 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4058000000000003e-05 2024-09-16T04:08:37.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.774 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.774 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.774 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099845+0000", 2024-09-16T04:08:37.775 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5729999999999997e-06 2024-09-16T04:08:37.775 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.775 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.775 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.775 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110509+0000", 2024-09-16T04:08:37.775 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010664339 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110550+0000", 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640999999999999e-05 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.777 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:37.777 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:37.777 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.579734844000001, 2024-09-16T04:08:37.777 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:37.777 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.777 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.777 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.777 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.778 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.778 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.778 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:37.778 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.778 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.778 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.778 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.779 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:37.780 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:37.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.780 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.780 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.780 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:37.780 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:37.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.780 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.780 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:37.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.782 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.578347430000001, 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:37.783 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.784 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.784 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.784 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:37.784 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.784 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.785 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.785 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:37.785 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:37.785 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.786 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.786 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.786 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:37.786 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:37.786 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.786 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.786 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.787 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:37.787 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:37.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.787 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.787 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.787 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:37.788 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:37.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.788 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.788 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.789 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:37.789 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:37.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.789 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.789 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.789 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:37.789 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:37.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.790 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.790 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.790 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:37.790 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:37.790 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.790 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.790 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.791 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:37.791 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:37.791 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.791 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.791 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.791 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.791 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.791 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.576080375, 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099172860000000009, 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:37.792 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.793 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.793 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:37.793 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.793 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.794 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107377+0000", 2024-09-16T04:08:37.794 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8900000000000001e-07 2024-09-16T04:08:37.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.794 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.794 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.795 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107437+0000", 2024-09-16T04:08:37.795 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0628000000000003e-05 2024-09-16T04:08:37.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.795 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.796 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.796 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107526+0000", 2024-09-16T04:08:37.796 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8873000000000001e-05 2024-09-16T04:08:37.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.796 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107535+0000", 2024-09-16T04:08:37.797 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0489999999999993e-06 2024-09-16T04:08:37.797 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.797 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.797 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.797 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111543+0000", 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040080810000000001 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111554+0000", 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0321e-05 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.798 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117271+0000", 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057176529999999996 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117294+0000", 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2391999999999998e-05 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=188, reqid=client.4440.0:379, at_version=47'56, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.231215972999999, 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011758437, 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:37.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:37.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:37.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:08:37.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452266+0000", 2024-09-16T04:08:37.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4151e-05 2024-09-16T04:08:37.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459768+0000", 2024-09-16T04:08:37.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075025179999999997 2024-09-16T04:08:37.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459778+0000", 2024-09-16T04:08:37.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0244999999999999e-05 2024-09-16T04:08:37.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459835+0000", 2024-09-16T04:08:37.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6394999999999998e-05 2024-09-16T04:08:37.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459845+0000", 2024-09-16T04:08:37.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0518e-05 2024-09-16T04:08:37.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463963+0000", 2024-09-16T04:08:37.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041182450000000004 2024-09-16T04:08:37.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463999+0000", 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5667999999999998e-05 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.219028399999999, 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.807 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:08:37.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:37.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:08:37.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:08:37.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:08:37.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:08:37.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:08:37.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:08:37.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:08:37.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:08:37.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:426 3.13s0 3:ce079411:::smithi01639990-2:d [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.175956074, 2024-09-16T04:08:37.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097940059999999992, 2024-09-16T04:08:37.815 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.815 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.815 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:37.815 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:37.815 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:37.816 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:37.816 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 426 2024-09-16T04:08:37.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.816 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.817 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.817 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:37.817 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.817 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.817 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.817 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4799999999999999e-07 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.818 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507505+0000", 2024-09-16T04:08:37.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0149999999999997e-06 2024-09-16T04:08:37.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.819 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.819 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.819 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507511+0000", 2024-09-16T04:08:37.819 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4540000000000004e-06 2024-09-16T04:08:37.819 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.819 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507517+0000", 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0749999999999997e-06 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.508922+0000", 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001405211 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.820 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.509012+0000", 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9776000000000002e-05 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.517295+0000", 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082831269999999995 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.822 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:37.822 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:37.822 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.2461584500000003, 2024-09-16T04:08:37.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019153804, 2024-09-16T04:08:37.822 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.822 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.822 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.822 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437299+0000", 2024-09-16T04:08:37.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437320+0000", 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1121999999999999e-05 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437388+0000", 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7889e-05 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437397+0000", 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4390000000000006e-06 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441658+0000", 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042607110000000004 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441667+0000", 2024-09-16T04:08:37.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456416+0000", 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014748493 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456452+0000", 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6159999999999999e-05 2024-09-16T04:08:37.826 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.2429687390000002, 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017174697999999999, 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440489+0000", 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.015e-06 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440509+0000", 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0338000000000001e-05 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440547+0000", 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440557+0000", 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7820000000000005e-06 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442068+0000", 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015112179999999999 2024-09-16T04:08:37.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.831 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.831 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442078+0000", 2024-09-16T04:08:37.831 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4620000000000002e-06 2024-09-16T04:08:37.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457639+0000", 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015560954 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457663+0000", 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4121000000000002e-05 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.832 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=82, reqid=osd.8.0:81, at_version=0'0, trim_to=0'0, roll_forward_to=57'51))", 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.2232444190000002, 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010694719, 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:37.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.834 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:37.834 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:37.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.834 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:37.834 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:37.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:37.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460213+0000", 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4999999999999999e-07 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460222+0000", 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2669999999999995e-06 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:37.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460651+0000", 2024-09-16T04:08:37.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00042901799999999998 2024-09-16T04:08:37.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:37.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.462020+0000", 2024-09-16T04:08:37.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013696680000000001 2024-09-16T04:08:37.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:37.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.466451+0000", 2024-09-16T04:08:37.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044306529999999997 2024-09-16T04:08:37.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:37.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.466460+0000", 2024-09-16T04:08:37.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:08:37.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.470859+0000", 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043990979999999997 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.470907+0000", 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7679000000000002e-05 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:37.839 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:37.840 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:37.841 INFO:tasks.rados.rados.0.smithi016.stdout:214: done (5 left) 2024-09-16T04:08:37.841 INFO:tasks.rados.rados.0.smithi016.stdout:216: done (4 left) 2024-09-16T04:08:37.841 INFO:tasks.rados.rados.0.smithi016.stdout:217: done (3 left) 2024-09-16T04:08:37.841 INFO:tasks.rados.rados.0.smithi016.stdout:221: done (2 left) 2024-09-16T04:08:37.841 INFO:tasks.rados.rados.0.smithi016.stdout:222: snap_remove snap 15 2024-09-16T04:08:37.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:37.843 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:37.839+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:37.942 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:37.940+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:37.952 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:37.952 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:37.952 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:37.952 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:37.961 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:38.042 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:38.041+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:38.063 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:38.063 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:38.064 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:38.064 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:38.064 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:38.072 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:38.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.es2.1 2024-09-16T04:08:38.175 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:38.173+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:38.178 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.071673218000001, 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073296819999999997, 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.179 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103493+0000", 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103494+0000", 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1399999999999997e-07 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103514+0000", 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9698999999999999e-05 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103592+0000", 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8378000000000001e-05 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103598+0000", 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9100000000000002e-06 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105790+0000", 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021916359999999998 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105798+0000", 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2490000000000001e-06 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110788+0000", 2024-09-16T04:08:38.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049902660000000001 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.110823+0000", 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5129999999999997e-05 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=34, reqid=client.4440.0:211, at_version=33'20, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.059456281999999, 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078650630000000003, 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115710+0000", 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115711+0000", 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2399999999999998e-07 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.115771+0000", 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9478999999999998e-05 2024-09-16T04:08:38.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119749+0000", 2024-09-16T04:08:38.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039784099999999999 2024-09-16T04:08:38.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119760+0000", 2024-09-16T04:08:38.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0664e-05 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119899+0000", 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013905300000000001 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119919+0000", 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9854999999999999e-05 2024-09-16T04:08:38.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123544+0000", 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0036256499999999998 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.123575+0000", 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1328000000000002e-05 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.189 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 33/27 ECSubWrite(tid=35, reqid=client.4440.0:212, at_version=33'21, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.056701159999999, 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077855709999999998, 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.190 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:38.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:38.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118466+0000", 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9999999999999996e-07 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.118551+0000", 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4878999999999998e-05 2024-09-16T04:08:38.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120595+0000", 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020432639999999999 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120604+0000", 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0250000000000008e-06 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122166+0000", 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0015625599999999999 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122175+0000", 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8470000000000002e-06 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126217+0000", 2024-09-16T04:08:38.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040415249999999998 2024-09-16T04:08:38.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.126251+0000", 2024-09-16T04:08:38.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4570999999999998e-05 2024-09-16T04:08:38.196 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.196 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.196 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.196 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:38.196 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:38.196 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.033473397000002, 2024-09-16T04:08:38.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:38.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:296 3.9s0 3:9fd10cc9:::smithi01639990-50:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:38.201 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.070593507000002, 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010626313, 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104575+0000", 2024-09-16T04:08:38.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1349999999999999e-06 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104586+0000", 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0603e-05 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104606+0000", 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9840999999999999e-05 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104615+0000", 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0189999999999995e-06 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104674+0000", 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9534999999999998e-05 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104680+0000", 2024-09-16T04:08:38.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9320000000000001e-06 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109134+0000", 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044534120000000003 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109195+0000", 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0983000000000002e-05 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109711+0000", 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051621600000000001 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.110753+0000", 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001042533 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:38.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115115+0000", 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043613030000000004 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115200+0000", 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4801000000000003e-05 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.726144135999998, 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085197069999999996, 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:38.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4000000000000001e-07 2024-09-16T04:08:38.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449160+0000", 2024-09-16T04:08:38.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013637999999999999 2024-09-16T04:08:38.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451237+0000", 2024-09-16T04:08:38.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020768179999999998 2024-09-16T04:08:38.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451246+0000", 2024-09-16T04:08:38.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:38.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451532+0000", 2024-09-16T04:08:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00028571800000000002 2024-09-16T04:08:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451542+0000", 2024-09-16T04:08:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0118999999999999e-05 2024-09-16T04:08:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457500+0000", 2024-09-16T04:08:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059576680000000002 2024-09-16T04:08:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457542+0000", 2024-09-16T04:08:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2716999999999998e-05 2024-09-16T04:08:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.715417789, 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:38.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:08:38.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:38.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:08:38.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:08:38.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:08:38.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:08:38.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:08:38.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:08:38.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:08:38.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:08:38.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:08:38.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:08:38.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:08:38.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:08:38.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:08:38.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:08:38.222 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.222 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.222 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.222 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:38.222 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:38.222 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.584875629999999, 2024-09-16T04:08:38.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:38.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:08:38.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:08:38.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:08:38.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:08:38.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:08:38.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:08:38.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:08:38.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:08:38.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:363 3.9s0 3:94de6fe1:::smithi01639990-3:head [stat,delete] snapc 8=[6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.735308969999998, 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085012569999999999, 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 363 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439860+0000", 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439869+0000", 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3170000000000008e-06 2024-09-16T04:08:38.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439888+0000", 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9293e-05 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439897+0000", 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439953+0000", 2024-09-16T04:08:38.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6317999999999998e-05 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439959+0000", 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3040000000000004e-06 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441596+0000", 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001637655 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441659+0000", 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2658999999999998e-05 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.442297+0000", 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063811800000000002 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.443562+0000", 2024-09-16T04:08:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001265361 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448304+0000", 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047413560000000004 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448359+0000", 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5189000000000003e-05 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.234 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.503953077, 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:38.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.343549018999999, 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.239 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:419 3.9s0 3:94de6fe1:::smithi01639990-3:head [append 0~786432] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.672654945, 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082147929999999997, 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.243 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 419 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502514+0000", 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.65e-06 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503628+0000", 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001113239 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503646+0000", 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7906000000000001e-05 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503653+0000", 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2339999999999997e-06 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503741+0000", 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8436999999999994e-05 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503820+0000", 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8751000000000005e-05 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505036+0000", 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001216017 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510097+0000", 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050614550000000003 2024-09-16T04:08:38.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510625+0000", 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000527667 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510727+0000", 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101437 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:420 3.9s0 3:94de6fe1:::smithi01639990-3:head [append 0~491520] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.670023044000001, 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089351980000000001, 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 420 2024-09-16T04:08:38.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:38.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505614+0000", 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00046960099999999999 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505627+0000", 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3275000000000001e-05 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505636+0000", 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2609999999999993e-06 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505739+0000", 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010322700000000001 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505847+0000", 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107929 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.506958+0000", 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011114580000000001 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510232+0000", 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032740080000000001 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.513941+0000", 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037089470000000002 2024-09-16T04:08:38.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514079+0000", 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000137744 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:422 3.9s0 3:94de6fe1:::smithi01639990-3:head [setxattr _header (56) in=7b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507590+0000", 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.667576809, 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007312804, 2024-09-16T04:08:38.254 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 422 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507590+0000", 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507590+0000", 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507591+0000", 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0800000000000004e-07 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507594+0000", 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5329999999999999e-06 2024-09-16T04:08:38.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507602+0000", 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5379999999999997e-06 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507607+0000", 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4500000000000003e-06 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.509353+0000", 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001745468 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.509454+0000", 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101718 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.509859+0000", 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040448799999999999 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.513811+0000", 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039521690000000002 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514806+0000", 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099465199999999995 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514903+0000", 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6979999999999996e-05 2024-09-16T04:08:38.259 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:423 3.9s0 3:94de6fe1:::smithi01639990-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.667550668000001, 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010412819, 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.260 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 423 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507617+0000", 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6599999999999996e-07 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507621+0000", 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9609999999999999e-06 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507628+0000", 2024-09-16T04:08:38.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3139999999999998e-06 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507637+0000", 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1840000000000005e-06 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510428+0000", 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027909060000000001 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510505+0000", 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6679000000000002e-05 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514954+0000", 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044494410000000002 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515004+0000", 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0244999999999998e-05 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518029+0000", 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030245229999999999 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:427 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507644+0000", 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.667522354000001, 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079136850000000002, 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.265 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 427 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507644+0000", 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507644+0000", 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507645+0000", 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4499999999999997e-07 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507649+0000", 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5789999999999997e-06 2024-09-16T04:08:38.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507655+0000", 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6729999999999997e-06 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507660+0000", 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0329999999999999e-06 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510813+0000", 2024-09-16T04:08:38.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031532259999999999 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510938+0000", 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000124757 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.511503+0000", 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00056476499999999995 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514248+0000", 2024-09-16T04:08:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002745272 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515498+0000", 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012494139999999999 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515558+0000", 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0520999999999999e-05 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:428 3.9s0 3:92dcccbe:::smithi01639990-16:head [rollback c] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.667488283000001, 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0081187840000000004, 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 428 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.271 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507679+0000", 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0900000000000002e-07 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507684+0000", 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3199999999999999e-06 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507691+0000", 2024-09-16T04:08:38.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1160000000000001e-06 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507697+0000", 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3129999999999998e-06 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512167+0000", 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044708880000000001 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512254+0000", 2024-09-16T04:08:38.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6329999999999995e-05 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512662+0000", 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040792999999999998 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514513+0000", 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018516019999999999 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:38.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515756+0000", 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001242976 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515797+0000", 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0899999999999998e-05 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:38.276 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.667444182000001, 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:08:38.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.6457311570000002, 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:08:38.282 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:08:38.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:08:38.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.6456789300000008, 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:08:38.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:08:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:08:38.293 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.293 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.293 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.293 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.293 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.293 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:38.295 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:38.295 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:38.291+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:38.395 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:38.393+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:38.438 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:38.438 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:38.438 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:38.438 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:38.447 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:38.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:08:38.495 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:38.494+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:222: done (2 left) 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:223: snap_create 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:218: finishing copy_from to smithi01639990-14 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 14 v 47 (ObjNum 94 snap 12 seq_num 94) dirty exists 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:219: finishing write tid 5 to smithi01639990-23 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:219: oid 23 updating version 0 to 21 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:219: oid 23 updating version 21 to 23 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:219: oid 23 version 23 is already newer than 20 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:219: oid 23 version 23 is already newer than 22 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 23 v 23 (ObjNum 104 snap 17 seq_num 104) dirty exists 2024-09-16T04:08:38.510 INFO:tasks.rados.rados.0.smithi016.stdout:219: left oid 23 (ObjNum 104 snap 17 seq_num 104) 2024-09-16T04:08:38.557 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:38.557 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:38.557 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:38.557 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:38.557 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:38.566 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:38.596 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:38.594+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:38.670 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:38.670 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:38.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:38.670 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 31.573679059, 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:38.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:38.675 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.676 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.676 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:38.676 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:38.676 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.568238586, 2024-09-16T04:08:38.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067064200000000003, 2024-09-16T04:08:38.677 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.677 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.677 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.677 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:38.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099748+0000", 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099749+0000", 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1299999999999999e-07 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099771+0000", 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2642000000000001e-05 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099811+0000", 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9406999999999997e-05 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.099819+0000", 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100401+0000", 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058156199999999999 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.100411+0000", 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7079999999999992e-06 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106408+0000", 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059972369999999999 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106454+0000", 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6384000000000002e-05 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.681 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.563791674000001, 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062253660000000004, 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104195+0000", 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104196+0000", 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7000000000000003e-07 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104218+0000", 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1974999999999999e-05 2024-09-16T04:08:38.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104245+0000", 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7489999999999999e-05 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104254+0000", 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104313+0000", 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9151000000000003e-05 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.104322+0000", 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110389+0000", 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.006067205 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110420+0000", 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1100999999999999e-05 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=164, reqid=osd.7.0:164, at_version=37'16, trim_to=0'0, roll_forward_to=35'12))", 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.557975152000001, 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062070839999999999, 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.686 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110011+0000", 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110012+0000", 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3999999999999997e-07 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110025+0000", 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3053e-05 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110784+0000", 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075944100000000002 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110793+0000", 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4880000000000007e-06 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110853+0000", 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0333e-05 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.110861+0000", 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2090000000000005e-06 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116184+0000", 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0053227389999999999 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.116218+0000", 2024-09-16T04:08:38.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4480999999999997e-05 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 37/27 ECSubWrite(tid=129, reqid=osd.3.0:129, at_version=37'43, trim_to=0'0, roll_forward_to=35'41))", 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.555209366, 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061976449999999999, 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112777+0000", 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112778+0000", 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3499999999999996e-07 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112797+0000", 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9578000000000001e-05 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112842+0000", 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5093999999999999e-05 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112850+0000", 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9759999999999995e-06 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112972+0000", 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000122033 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112982+0000", 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5789999999999993e-06 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118944+0000", 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005961816 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.118975+0000", 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0933999999999999e-05 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.696 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=155, reqid=osd.3.0:155, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.228346253000002, 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073085160000000001, 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1299999999999999e-07 2024-09-16T04:08:38.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439663+0000", 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3030000000000001e-05 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439705+0000", 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1078999999999998e-05 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439714+0000", 2024-09-16T04:08:38.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441461+0000", 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017466350000000001 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441475+0000", 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4404e-05 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446911+0000", 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054361430000000001 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446949+0000", 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7465000000000001e-05 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:38.701 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.224267485999999, 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068399539999999997, 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:38.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443720+0000", 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9500000000000001e-07 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444146+0000", 2024-09-16T04:08:38.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042652899999999998 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444178+0000", 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1412999999999999e-05 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444185+0000", 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4270000000000002e-06 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444247+0000", 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1760000000000005e-05 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444254+0000", 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9260000000000003e-06 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450519+0000", 2024-09-16T04:08:38.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062649960000000001 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450559+0000", 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0108000000000003e-05 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s5 42/27 ECSubWrite(tid=40, reqid=osd.0.0:40, at_version=42'11, trim_to=0'0, roll_forward_to=41'9))", 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:38.706 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.224093478, 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067397389999999998, 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0700000000000001e-07 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443914+0000", 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1274e-05 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443937+0000", 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2991999999999999e-05 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443945+0000", 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4780000000000002e-06 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444021+0000", 2024-09-16T04:08:38.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5814000000000003e-05 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444033+0000", 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2322e-05 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450586+0000", 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065529869999999997 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450633+0000", 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6764999999999997e-05 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.238162088999999, 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.711 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:08:38.714 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.778211861999999, 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:08:38.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.719 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:08:38.720 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:38.720 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.454699257, 2024-09-16T04:08:38.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:08:38.720 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.720 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.720 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:38.720 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:08:38.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:08:38.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.196915399, 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078488229999999996, 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471072+0000", 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0279999999999999e-06 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471418+0000", 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034541800000000001 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471457+0000", 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471492+0000", 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471798+0000", 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030615599999999997 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471812+0000", 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.448e-05 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478881+0000", 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070685549999999998 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478920+0000", 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9072000000000001e-05 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.728 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 53/27 ECSubWrite(tid=103, reqid=osd.5.0:103, at_version=53'29, trim_to=0'0, roll_forward_to=52'27))", 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.180358740999999, 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068898910000000004, 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.22e-07 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487655+0000", 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.669e-05 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487696+0000", 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0930000000000003e-05 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487708+0000", 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2699e-05 2024-09-16T04:08:38.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487808+0000", 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9557999999999998e-05 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487819+0000", 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1297999999999999e-05 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.494471+0000", 2024-09-16T04:08:38.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0066521230000000002 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.494518+0000", 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6371e-05 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 53/46 ECSubWrite(tid=213, reqid=osd.3.0:213, at_version=53'65, trim_to=0'0, roll_forward_to=52'63))", 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:38.733 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.177587728000001, 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070799060000000004, 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490424+0000", 2024-09-16T04:08:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5196999999999999e-05 2024-09-16T04:08:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490482+0000", 2024-09-16T04:08:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7442000000000003e-05 2024-09-16T04:08:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490499+0000", 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7467999999999999e-05 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490695+0000", 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000195897 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490707+0000", 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2221999999999999e-05 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.497426+0000", 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067186169999999996 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.497479+0000", 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2876e-05 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.738 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 53/46 ECSubWrite(tid=216, reqid=osd.3.0:213, at_version=0'0, trim_to=0'0, roll_forward_to=53'65))", 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.498161+0000", 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.169825612, 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059890799999999999, 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498161+0000", 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498161+0000", 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498162+0000", 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9299999999999996e-07 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498174+0000", 2024-09-16T04:08:38.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2391e-05 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498188+0000", 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3835000000000001e-05 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498194+0000", 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9869999999999996e-06 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498264+0000", 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0053000000000002e-05 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498276+0000", 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2251e-05 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.504086+0000", 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005809982 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.504150+0000", 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3688000000000004e-05 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.15s7 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:38.743 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.438296+0000", 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.2296902020000005, 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064265700000000004, 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438296+0000", 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438296+0000", 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438297+0000", 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.47e-07 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438328+0000", 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1115000000000003e-05 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438376+0000", 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7774e-05 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438385+0000", 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6719999999999993e-06 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438449+0000", 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4206000000000002e-05 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.438460+0000", 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0862e-05 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.444681+0000", 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062210820000000002 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.444723+0000", 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2011999999999997e-05 2024-09-16T04:08:38.747 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.748 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.748 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.748 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:38.748 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:38.748 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.227800931, 2024-09-16T04:08:38.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084259260000000002, 2024-09-16T04:08:38.748 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.91e-07 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440211+0000", 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4978999999999999e-05 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440257+0000", 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6569000000000001e-05 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440268+0000", 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1056000000000001e-05 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440348+0000", 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9795000000000004e-05 2024-09-16T04:08:38.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440364+0000", 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5996999999999999e-05 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448570+0000", 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082059590000000005 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448611+0000", 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.138e-05 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.bs7 57/56 [PushOp(3:d683cb60:::smithi01639990-49:head, version: 53'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d683cb60:::smithi01639990-49:head@53'38, size: 2310144, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.1072187600000003, 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.043702234999999999, 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.753 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560769+0000", 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0049999999999999e-06 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.561301+0000", 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053276500000000004 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603955+0000", 2024-09-16T04:08:38.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042653605999999997 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603967+0000", 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2065000000000001e-05 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604063+0000", 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5599000000000004e-05 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604470+0000", 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040719500000000001 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=252, reqid=osd.3.0:252, at_version=57'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.7584926200000002, 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27497091299999998, 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.757 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909517+0000", 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2886000000000001e-05 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183847+0000", 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27432981400000001 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183853+0000", 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2929999999999997e-06 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183902+0000", 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9008999999999998e-05 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183911+0000", 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:08:38.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184435+0000", 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052493699999999995 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184465+0000", 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9292e-05 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=253, reqid=osd.3.0:253, at_version=57'67, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.7578645890000004, 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50851210899999999, 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910123+0000", 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910136+0000", 2024-09-16T04:08:38.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.331e-05 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416910+0000", 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50677369000000005 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416919+0000", 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8290000000000003e-06 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416975+0000", 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5983999999999998e-05 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416985+0000", 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418599+0000", 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001614515 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:38.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:38.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:38.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418634+0000", 2024-09-16T04:08:38.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4588999999999997e-05 2024-09-16T04:08:38.766 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.766 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.766 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.766 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:38.766 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:38.766 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:38.768 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:38.768 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:38.765+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:38.868 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:38.866+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:38.913 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:38.913 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:38.913 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:38.919 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:38.969 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:38.968+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:39.062 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:39.062 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:39.062 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:39.063 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:39.063 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:39.070 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:39.071 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:39.067+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:39.171 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:39.169+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:39.216 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:39.216 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:39.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:39.216 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:39.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.216 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:39.216 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:39.216 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.110274232999998, 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083578109999999997, 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102824+0000", 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4899999999999995e-07 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102859+0000", 2024-09-16T04:08:39.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4805999999999998e-05 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102891+0000", 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2394000000000002e-05 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102898+0000", 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.204e-06 2024-09-16T04:08:39.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103849+0000", 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00095089899999999995 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103858+0000", 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3480000000000004e-06 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111149+0000", 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072911809999999999 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111181+0000", 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2329999999999997e-05 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.222 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:39.222 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.105289+0000", 2024-09-16T04:08:39.222 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.1078084, 2024-09-16T04:08:39.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065227080000000003, 2024-09-16T04:08:39.222 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.222 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.222 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.222 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105289+0000", 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105289+0000", 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.105290+0000", 2024-09-16T04:08:39.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.99e-07 2024-09-16T04:08:39.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106024+0000", 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00073392599999999998 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106050+0000", 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6239000000000001e-05 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106058+0000", 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.1429999999999998e-06 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106110+0000", 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1788e-05 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.106118+0000", 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.225e-06 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111780+0000", 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056617070000000002 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111812+0000", 2024-09-16T04:08:39.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1980999999999998e-05 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs3 37/27 ECSubWrite(tid=74, reqid=osd.5.0:72, at_version=0'0, trim_to=0'0, roll_forward_to=37'22))", 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:13.112863+0000", 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.100234415999999, 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062791490000000004, 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.228 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.229 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112863+0000", 2024-09-16T04:08:39.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112863+0000", 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112864+0000", 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4900000000000004e-07 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112875+0000", 2024-09-16T04:08:39.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1112e-05 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112905+0000", 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0142999999999999e-05 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112913+0000", 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9969999999999998e-06 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112967+0000", 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3928999999999998e-05 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.112976+0000", 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.003e-06 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119105+0000", 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0061291360000000003 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:13.119142+0000", 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6879999999999999e-05 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.233 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/27 ECSubWrite(tid=155, reqid=osd.3.0:155, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439852+0000", 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.773245619000001, 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070272690000000001, 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439852+0000", 2024-09-16T04:08:39.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439852+0000", 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439852+0000", 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0200000000000001e-07 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439882+0000", 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0241999999999999e-05 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439947+0000", 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4238000000000004e-05 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439954+0000", 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7360000000000001e-06 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440364+0000", 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040977599999999999 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.440374+0000", 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5629999999999998e-06 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446824+0000", 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064499730000000003 2024-09-16T04:08:39.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446879+0000", 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5538999999999998e-05 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443220+0000", 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.769877623999999, 2024-09-16T04:08:39.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064092510000000004, 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443220+0000", 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443220+0000", 2024-09-16T04:08:39.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443221+0000", 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9999999999999995e-07 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443544+0000", 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00032279200000000002 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443573+0000", 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9546000000000001e-05 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443580+0000", 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4849999999999997e-06 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443642+0000", 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2081000000000004e-05 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443652+0000", 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0209e-05 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449580+0000", 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059278250000000003 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449629+0000", 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9313e-05 2024-09-16T04:08:39.242 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.450278+0000", 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.762819749999998, 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007464432, 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450278+0000", 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450278+0000", 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450279+0000", 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.031e-06 2024-09-16T04:08:39.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450548+0000", 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000269659 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450626+0000", 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7577000000000006e-05 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450635+0000", 2024-09-16T04:08:39.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2920000000000001e-06 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450868+0000", 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00023291100000000001 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450879+0000", 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0988999999999999e-05 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457700+0000", 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068211620000000004 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457742+0000", 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1811000000000003e-05 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 48/46 ECSubWrite(tid=190, reqid=osd.3.0:190, at_version=48'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:39.247 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.747453159999999, 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088361580000000002, 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465645+0000", 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4399999999999999e-07 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465670+0000", 2024-09-16T04:08:39.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4875000000000002e-05 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465791+0000", 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012066900000000001 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465800+0000", 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4539999999999996e-06 2024-09-16T04:08:39.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465872+0000", 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2127999999999998e-05 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465881+0000", 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3129999999999999e-06 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474449+0000", 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085683760000000008 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474480+0000", 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1498999999999998e-05 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.475668+0000", 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.737429862000001, 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084809299999999994, 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475668+0000", 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475668+0000", 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475668+0000", 2024-09-16T04:08:39.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3700000000000001e-07 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475747+0000", 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9559000000000002e-05 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476148+0000", 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040095300000000001 2024-09-16T04:08:39.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476157+0000", 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9870000000000005e-06 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476859+0000", 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00070109199999999995 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476868+0000", 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1689999999999998e-06 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484115+0000", 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072471239999999998 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484149+0000", 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3708999999999999e-05 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 48/46 ECSubWrite(tid=58, reqid=client.4440.0:386, at_version=48'37, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.477170+0000", 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.735927569999999, 2024-09-16T04:08:39.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0069770179999999998, 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477170+0000", 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477170+0000", 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477171+0000", 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0070000000000001e-06 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477192+0000", 2024-09-16T04:08:39.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1458e-05 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477241+0000", 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8829999999999998e-05 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477251+0000", 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4590000000000004e-06 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477364+0000", 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000113713 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477374+0000", 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6549999999999999e-06 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484104+0000", 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067301820000000004 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484147+0000", 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2713999999999998e-05 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 52/46 ECSubWrite(tid=210, reqid=client.4440.0:434, at_version=52'63, trim_to=0'0, roll_forward_to=50'60))", 2024-09-16T04:08:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.695330633999999, 2024-09-16T04:08:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83621405000000004, 2024-09-16T04:08:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8200000000000001e-07 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517790+0000", 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2540000000000001e-05 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352789+0000", 2024-09-16T04:08:39.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83499984500000002 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352804+0000", 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4569999999999999e-05 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352928+0000", 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012396799999999999 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352943+0000", 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353943+0000", 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099977500000000006 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353981+0000", 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7700999999999997e-05 2024-09-16T04:08:39.268 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 53/46 ECSubWrite(tid=64, reqid=client.4440.0:443, at_version=53'44, trim_to=0'0, roll_forward_to=50'42))", 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.499624+0000", 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.713473386, 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0069932299999999996, 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499624+0000", 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499624+0000", 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499624+0000", 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3400000000000001e-07 2024-09-16T04:08:39.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499683+0000", 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8374e-05 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499705+0000", 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.213e-05 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499713+0000", 2024-09-16T04:08:39.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3100000000000001e-06 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499802+0000", 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9081999999999999e-05 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499813+0000", 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0477e-05 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506578+0000", 2024-09-16T04:08:39.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067654880000000001 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506617+0000", 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9235e-05 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=217, reqid=client.4440.0:449, at_version=53'58, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.701820597999999, 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0096255799999999999, 2024-09-16T04:08:39.273 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511278+0000", 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.068e-06 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511623+0000", 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034493499999999999 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515041+0000", 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034178889999999999 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515048+0000", 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3479999999999998e-06 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515115+0000", 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6469000000000003e-05 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515126+0000", 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1187e-05 2024-09-16T04:08:39.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520872+0000", 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057460280000000002 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520902+0000", 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0655999999999999e-05 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=218, reqid=client.4440.0:450, at_version=53'59, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.698476023, 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089217440000000005, 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:39.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514623+0000", 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2300000000000001e-06 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515021+0000", 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039790399999999998 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519384+0000", 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043635380000000001 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519391+0000", 2024-09-16T04:08:39.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9709999999999998e-06 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519441+0000", 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9493999999999997e-05 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519449+0000", 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:08:39.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.523504+0000", 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040551220000000004 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.523543+0000", 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9271999999999999e-05 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=220, reqid=client.4440.0:452, at_version=53'61, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.693327081, 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51827822899999998, 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519771+0000", 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.61e-07 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519788+0000", 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6699999999999999e-05 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036882+0000", 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51709414300000001 2024-09-16T04:08:39.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036895+0000", 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3363999999999999e-05 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036964+0000", 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8734999999999998e-05 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036973+0000", 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0289999999999994e-06 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038014+0000", 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010410529999999999 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038049+0000", 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4443999999999998e-05 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs3 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.446789+0000", 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.7663079699999997, 2024-09-16T04:08:39.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079296090000000007, 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.446789+0000", 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.446789+0000", 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.446790+0000", 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.08e-07 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.446827+0000", 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7766999999999999e-05 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.447541+0000", 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071359099999999996 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.447550+0000", 2024-09-16T04:08:39.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8170000000000005e-06 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.449555+0000", 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020049859999999998 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.449565+0000", 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0003999999999999e-05 2024-09-16T04:08:39.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.454688+0000", 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051229589999999998 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.454719+0000", 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1377000000000001e-05 2024-09-16T04:08:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:524 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~589824] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.623355+0000", 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.589742126, 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070847519999999997, 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.294 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 524 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.623355+0000", 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.623355+0000", 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.623358+0000", 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4880000000000001e-06 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.624250+0000", 2024-09-16T04:08:39.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00089222699999999999 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.624263+0000", 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3305999999999999e-05 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.624270+0000", 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1500000000000004e-06 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.297 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.624327+0000", 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7543000000000001e-05 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.624387+0000", 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0349000000000001e-05 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.625613+0000", 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001225863 2024-09-16T04:08:39.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.627431+0000", 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018175629999999999 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630357+0000", 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029266380000000001 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630440+0000", 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2625000000000003e-05 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:525 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~540672] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.626881+0000", 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.5862168210000001, 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076914009999999996, 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.300 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 525 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.626881+0000", 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.626881+0000", 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.626882+0000", 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.798e-06 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628281+0000", 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0013982829999999999 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628293+0000", 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2170999999999999e-05 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628299+0000", 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.088e-06 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628341+0000", 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2449e-05 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628454+0000", 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011251699999999999 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.629546+0000", 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001091828 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630981+0000", 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014352449999999999 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634515+0000", 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003533694 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634572+0000", 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7327999999999997e-05 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:526 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~622592] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.582530693, 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01502261, 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.306 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 526 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630568+0000", 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5680000000000001e-06 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631353+0000", 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078490700000000005 2024-09-16T04:08:39.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631367+0000", 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3441000000000001e-05 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631375+0000", 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8809999999999996e-06 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631465+0000", 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0673000000000004e-05 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631565+0000", 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9753000000000005e-05 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.632641+0000", 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001076097 2024-09-16T04:08:39.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634418+0000", 2024-09-16T04:08:39.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017769680000000001 2024-09-16T04:08:39.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645537+0000", 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011119393999999999 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645589+0000", 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1928000000000002e-05 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.312 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:527 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.58169362, 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015053211, 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 527 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631405+0000", 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.08e-06 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.314 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631411+0000", 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9009999999999999e-06 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631419+0000", 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631426+0000", 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4999999999999996e-06 2024-09-16T04:08:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633558+0000", 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002131796 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633663+0000", 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105833 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634106+0000", 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00044296899999999998 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.636380+0000", 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022740849999999999 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646383+0000", 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01000238 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646457+0000", 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4258000000000006e-05 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:528 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:39.318 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.581619039, 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65958352099999995, 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 528 2024-09-16T04:08:39.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631479+0000", 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9599999999999998e-07 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631483+0000", 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8160000000000004e-06 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631494+0000", 2024-09-16T04:08:39.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0407999999999999e-05 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631501+0000", 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2269999999999996e-06 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634655+0000", 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031544339999999998 2024-09-16T04:08:39.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634732+0000", 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6495999999999994e-05 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646864+0000", 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012131807999999999 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.323 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646965+0000", 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010173 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.291062+0000", 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64409670600000002 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.324 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.325 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.325 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:39.326 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:39.327 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:39.324+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:39.427 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:39.425+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:39.431 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:39.431 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:39.431 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:39.431 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:39.440 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:39.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:39.512 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-16T04:08:39.516 INFO:tasks.rados.rados.0.smithi016.stdout:218: done (2 left) 2024-09-16T04:08:39.517 INFO:tasks.rados.rados.0.smithi016.stdout:219: done (1 left) 2024-09-16T04:08:39.517 INFO:tasks.rados.rados.0.smithi016.stdout:223: done (0 left) 2024-09-16T04:08:39.517 INFO:tasks.rados.rados.0.smithi016.stdout:224: rmattr oid 16 current snap is 18 2024-09-16T04:08:39.517 INFO:tasks.rados.rados.0.smithi016.stdout:224: done (0 left) 2024-09-16T04:08:39.517 INFO:tasks.rados.rados.0.smithi016.stdout:225: read oid 11 snap -1 2024-09-16T04:08:39.517 INFO:tasks.rados.rados.0.smithi016.stdout:225: expect (ObjNum 71 snap 3 seq_num 71) 2024-09-16T04:08:39.517 INFO:tasks.rados.rados.0.smithi016.stdout:226: copy_from oid 15 from oid 49 current snap is 18 2024-09-16T04:08:39.517 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 50 current snap is 18 2024-09-16T04:08:39.517 INFO:tasks.rados.rados.0.smithi016.stdout:227: seq_num 106 ranges {0=622592,622592=442368,1064960=524288,1589248=196608} 2024-09-16T04:08:39.519 INFO:tasks.rados.rados.0.smithi016.stdout:227: writing smithi01639990-50 from 0 to 622592 tid 1 2024-09-16T04:08:39.523 INFO:tasks.rados.rados.0.smithi016.stdout:227: writing smithi01639990-50 from 622592 to 1064960 tid 2 2024-09-16T04:08:39.526 INFO:tasks.rados.rados.0.smithi016.stdout:227: writing smithi01639990-50 from 1064960 to 1589248 tid 3 2024-09-16T04:08:39.529 INFO:tasks.rados.rados.0.smithi016.stdout:227: writing smithi01639990-50 from 1589248 to 1785856 tid 4 2024-09-16T04:08:39.529 INFO:tasks.rados.rados.0.smithi016.stdout:228: read oid 43 snap -1 2024-09-16T04:08:39.529 INFO:tasks.rados.rados.0.smithi016.stdout:228: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:08:39.529 INFO:tasks.rados.rados.0.smithi016.stdout:229: read oid 33 snap -1 2024-09-16T04:08:39.529 INFO:tasks.rados.rados.0.smithi016.stdout:229: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:08:39.529 INFO:tasks.rados.rados.0.smithi016.stdout:230: rollback oid 28 current snap is 18 2024-09-16T04:08:39.529 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 28 to 17 2024-09-16T04:08:39.529 INFO:tasks.rados.rados.0.smithi016.stdout:231: rmattr oid 20 current snap is 18 2024-09-16T04:08:39.529 INFO:tasks.rados.rados.0.smithi016.stdout:231: done (6 left) 2024-09-16T04:08:39.530 INFO:tasks.rados.rados.0.smithi016.stdout:232: delete oid 27 current snap is 18 2024-09-16T04:08:39.530 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:39.527+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:39.530 INFO:tasks.rados.rados.0.smithi016.stdout:230: finishing rollback tid 1 to smithi01639990-28 2024-09-16T04:08:39.530 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 28 v 0 (ObjNum 33 snap 0 seq_num 3719738104) dirty dne 2024-09-16T04:08:39.535 INFO:tasks.rados.rados.0.smithi016.stdout:228: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:08:39.550 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:39.550 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:39.550 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:39.550 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:39.550 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:39.555 INFO:tasks.rados.rados.0.smithi016.stdout:229: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T04:08:39.559 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:39.569 INFO:tasks.rados.rados.0.smithi016.stdout:227: finishing write tid 1 to smithi01639990-50 2024-09-16T04:08:39.569 INFO:tasks.rados.rados.0.smithi016.stdout:227: finishing write tid 2 to smithi01639990-50 2024-09-16T04:08:39.569 INFO:tasks.rados.rados.0.smithi016.stdout:227: finishing write tid 3 to smithi01639990-50 2024-09-16T04:08:39.569 INFO:tasks.rados.rados.0.smithi016.stdout:227: finishing write tid 4 to smithi01639990-50 2024-09-16T04:08:39.570 INFO:tasks.rados.rados.0.smithi016.stdout:226: finishing copy_from racing read to smithi01639990-15 2024-09-16T04:08:39.570 INFO:tasks.rados.rados.0.smithi016.stdout:226: finishing copy_from to smithi01639990-15 2024-09-16T04:08:39.570 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 15 v 35 (ObjNum 88 snap 11 seq_num 88) dirty exists 2024-09-16T04:08:39.621 INFO:teuthology.orchestra.run.smithi016.stdout:get_pg_num_history pg_num_history pg_num_history(e54 pg_nums {1={19=1},2={20=8},3={27=16,46=26}} deleted_pools ) 2024-09-16T04:08:39.621 INFO:teuthology.orchestra.run.smithi016.stdout:Importing pgid 3.es2 2024-09-16T04:08:39.622 INFO:teuthology.orchestra.run.smithi016.stdout:Write 2#3:7f241f48:::smithi01639990-30:head# 2024-09-16T04:08:39.622 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 3=[]:{} 2024-09-16T04:08:39.622 INFO:teuthology.orchestra.run.smithi016.stdout:write_pg epoch 53 info 3.es2( v 34'8 (0'0,34'8] local-lis/les=27/28 n=1 ec=27/27 lis/c=27/27 les/c/f=28/28/0 sis=27) 2024-09-16T04:08:39.623 INFO:teuthology.orchestra.run.smithi016.stdout:Import successful 2024-09-16T04:08:39.626 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T04:08:39.630 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:39.627+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:39.648 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:124 1.0 e22/19 1:30306672:devicehealth::main.db.0000000000000000:head v 22'86, mlcod=21'84)", 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 47.231561556000003, 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020095396000000001, 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.665 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430782+0000", 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9699999999999996e-07 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430789+0000", 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4660000000000002e-06 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450028+0000", 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019238816999999998 2024-09-16T04:08:39.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450042+0000", 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4487e-05 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450169+0000", 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126787 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450188+0000", 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450816+0000", 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062741399999999997 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450877+0000", 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0782999999999997e-05 2024-09-16T04:08:39.669 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:197 3.es0 3:7f241f48:::smithi01639990-30:head [copy-from ver 23] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 31.452352208000001, 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.023242381999999999, 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.670 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 197 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6499999999999999e-07 2024-09-16T04:08:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209995+0000", 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.388e-06 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210002+0000", 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5209999999999997e-06 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210008+0000", 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8780000000000003e-06 2024-09-16T04:08:39.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8421999999999999e-05 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210099+0000", 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2469999999999997e-05 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227450+0000", 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017351235 2024-09-16T04:08:39.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232352+0000", 2024-09-16T04:08:39.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049017239999999997 2024-09-16T04:08:39.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233168+0000", 2024-09-16T04:08:39.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081616600000000001 2024-09-16T04:08:39.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233233+0000", 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5012999999999996e-05 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:198 3.es0 3:7f241f48:::smithi01639990-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 31.452321899000001, 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019673758, 2024-09-16T04:08:39.677 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 198 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3099999999999998e-07 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210024+0000", 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2620000000000001e-06 2024-09-16T04:08:39.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210030+0000", 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9020000000000002e-06 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210037+0000", 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1480000000000003e-06 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210227+0000", 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00019055499999999999 2024-09-16T04:08:39.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210269+0000", 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2246999999999997e-05 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229480+0000", 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019210896000000002 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229587+0000", 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107026 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229695+0000", 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107291 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:213 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.546243394000001, 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53275775299999995, 2024-09-16T04:08:39.682 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 213 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116100+0000", 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116103+0000", 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.174e-06 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116109+0000", 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4330000000000003e-06 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116114+0000", 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1399999999999999e-06 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119298+0000", 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031846729999999998 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119365+0000", 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6484999999999998e-05 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119675+0000", 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00031021999999999999 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122365+0000", 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002689851 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648785+0000", 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52641971300000001 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648857+0000", 2024-09-16T04:08:39.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2408000000000002e-05 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:214 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.546220718000001, 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53416417900000002, 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:39.688 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 214 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2100000000000002e-07 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116125+0000", 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2649999999999999e-06 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116132+0000", 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5190000000000003e-06 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116137+0000", 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9869999999999999e-06 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120036+0000", 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038984979999999998 2024-09-16T04:08:39.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120118+0000", 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2312999999999996e-05 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649191+0000", 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52907320199999996 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649279+0000", 2024-09-16T04:08:39.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7585999999999998e-05 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.650286+0000", 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010073879999999999 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "replica_scrub(pg: 3.4s1,from:0'0,to:37'17,epoch:47/46,start:3:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.804419181, 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16799329900000001, 2024-09-16T04:08:39.693 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9400000000000005e-07 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857929+0000", 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4719999999999997e-06 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025793+0000", 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16786378299999999 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025803+0000", 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9629999999999993e-06 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025875+0000", 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2914e-05 2024-09-16T04:08:39.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025887+0000", 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1345999999999999e-05 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025917+0000", 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0127000000000001e-05 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.4s1,RELEASE,e:47,from:1(0),reservation_nonce:0)", 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.635274205999998, 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26135687200000002, 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2800000000000005e-07 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027075+0000", 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3859999999999998e-06 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288302+0000", 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26122710999999998 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288308+0000", 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6189999999999999e-06 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288392+0000", 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4549000000000007e-05 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288414+0000", 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1576000000000001e-05 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288425+0000", 2024-09-16T04:08:39.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1704e-05 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:385 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.196271933999999, 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01911771, 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.702 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 385 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:39.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466073+0000", 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4389999999999999e-06 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466084+0000", 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1157e-05 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466101+0000", 2024-09-16T04:08:39.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6334999999999999e-05 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466111+0000", 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466255+0000", 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014411 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:39.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466263+0000", 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.858e-06 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474564+0000", 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083014019999999994 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474734+0000", 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016996599999999999 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475786+0000", 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010515909999999999 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.482124+0000", 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063382530000000003 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485117+0000", 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029928120000000001 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485189+0000", 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1868999999999999e-05 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:386 3.8s0 3:17b72cd6:::smithi01639990-47:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.196200201, 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019422101000000001, 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 386 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8100000000000002e-07 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466147+0000", 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6559999999999998e-06 2024-09-16T04:08:39.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466155+0000", 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6240000000000002e-06 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466162+0000", 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4850000000000003e-06 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466343+0000", 2024-09-16T04:08:39.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000181077 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466349+0000", 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1229999999999997e-06 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476407+0000", 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010057885000000001 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476516+0000", 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108738 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477098+0000", 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000582024 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484050+0000", 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0069524950000000004 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485519+0000", 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014691280000000001 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485565+0000", 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5185000000000002e-05 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.714 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:387 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.196172099999998, 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51761721800000005, 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 387 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6399999999999998e-07 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466173+0000", 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.396e-06 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466179+0000", 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9290000000000003e-06 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466184+0000", 2024-09-16T04:08:39.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9760000000000003e-06 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466397+0000", 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021269400000000001 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466406+0000", 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0769999999999997e-06 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479302+0000", 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012896014000000001 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479415+0000", 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112767 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479914+0000", 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00049861199999999995 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484063+0000", 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004149158 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983716+0000", 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49965376500000003 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983788+0000", 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1465999999999996e-05 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.720 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.721 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/46 ECSubWriteReply(tid=59, last_complete=48'38, committed=1, applied=1))", 2024-09-16T04:08:39.721 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:39.721 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.178131088000001, 2024-09-16T04:08:39.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50006588100000005, 2024-09-16T04:08:39.721 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.721 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.721 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2499999999999999e-07 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484214+0000", 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6440000000000001e-06 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983573+0000", 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49935939200000001 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983581+0000", 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983628+0000", 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7642999999999998e-05 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.984278+0000", 2024-09-16T04:08:39.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00064939900000000005 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "rep_scrubmap(3.8s0 e46 from shard 10(6))", 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.676029796, 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21666559099999999, 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986314+0000", 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5600000000000005e-07 2024-09-16T04:08:39.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986319+0000", 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0950000000000003e-06 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202862+0000", 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21654300900000001 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202872+0000", 2024-09-16T04:08:39.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0077e-05 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202931+0000", 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9568000000000003e-05 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202942+0000", 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0139999999999999e-05 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202979+0000", 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6946000000000002e-05 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s3 53/48 ECSubWrite(tid=39, reqid=osd.4.0:39, at_version=53'24, trim_to=0'0, roll_forward_to=51'22))", 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.173405008, 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017845360000000001, 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.729 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2000000000000001e-07 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488964+0000", 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6418e-05 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502624+0000", 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013659439000000001 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502636+0000", 2024-09-16T04:08:39.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2401999999999999e-05 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502702+0000", 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6006999999999993e-05 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502712+0000", 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8870000000000004e-06 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506740+0000", 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040279319999999997 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506783+0000", 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3055e-05 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:443 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~425984] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.495031+0000", 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.167311478, 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011905696, 2024-09-16T04:08:39.733 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 443 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.495031+0000", 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.495031+0000", 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.495038+0000", 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.443e-06 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498278+0000", 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032401050000000001 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498294+0000", 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6531000000000001e-05 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498304+0000", 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3130000000000004e-06 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498378+0000", 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4790999999999996e-05 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498451+0000", 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2082000000000005e-05 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499529+0000", 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001078648 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506272+0000", 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067422960000000001 2024-09-16T04:08:39.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506846+0000", 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00057419700000000003 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506937+0000", 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1290000000000002e-05 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:446 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.158803348999999, 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67862900900000001, 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 446 2024-09-16T04:08:39.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503541+0000", 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3680000000000001e-06 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503545+0000", 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.561e-06 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503553+0000", 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8939999999999993e-06 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503559+0000", 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.224e-06 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505239+0000", 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001679739 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505303+0000", 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4219000000000003e-05 2024-09-16T04:08:39.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505673+0000", 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000369374 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.507082+0000", 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014090330000000001 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182055+0000", 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67497357700000005 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182168+0000", 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011302 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:447 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.158773843000001, 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.680085987, 2024-09-16T04:08:39.744 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 447 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:39.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503570+0000", 2024-09-16T04:08:39.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.44e-07 2024-09-16T04:08:39.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503576+0000", 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4559999999999997e-06 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503602+0000", 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6264000000000001e-05 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503610+0000", 2024-09-16T04:08:39.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4109999999999999e-06 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505945+0000", 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023353390000000001 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506004+0000", 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9410999999999998e-05 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182603+0000", 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67659811199999997 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182695+0000", 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1951000000000001e-05 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.183655+0000", 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00096039900000000002 2024-09-16T04:08:39.751 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.153701518, 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.299096274, 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508642+0000", 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5400000000000002e-07 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508645+0000", 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9249999999999999e-06 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807520+0000", 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.29887539899999999 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807534+0000", 2024-09-16T04:08:39.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3643e-05 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807681+0000", 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014696699999999999 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807738+0000", 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6885999999999997e-05 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.755 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.153320053, 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67354323000000005, 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:39.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9500000000000002e-07 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509027+0000", 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4429999999999999e-06 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181881+0000", 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67285377899999999 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181891+0000", 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0479000000000001e-05 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181961+0000", 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0370000000000006e-05 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182566+0000", 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060456400000000003 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.759 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:520 3.es0 3:7f241f48:::smithi01639990-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.620430+0000", 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.041913036, 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013388690999999999, 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 520 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620430+0000", 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620430+0000", 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620431+0000", 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4399999999999998e-07 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620435+0000", 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1529999999999999e-06 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620445+0000", 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0458999999999999e-05 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620451+0000", 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0519999999999998e-06 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.623070+0000", 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026187659999999998 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.623129+0000", 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8658e-05 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.629860+0000", 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067308730000000001 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.629972+0000", 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011219599999999999 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633818+0000", 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00384659 2024-09-16T04:08:39.764 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.4s1 62/56 ECSubRead(tid=8, to_read={3:26eb5942:::smithi01639990-23:head=0,4096,0}, subchunks={3:26eb5942:::smithi01639990-23:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 2.014689223, 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64312231099999995, 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:39.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2500000000000004e-07 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:39.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647678+0000", 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3421000000000002e-05 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290437+0000", 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64275884900000002 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290450+0000", 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2858e-05 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290545+0000", 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5463000000000004e-05 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290776+0000", 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000230895 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:39.768 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:39.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:39.770 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:39.770 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:39.771 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:39.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:39.771 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:39.772 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:39.772 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:39.772 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:39.773 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:39.773 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:39.768+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:39.775 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:39.776 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:39.797 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:39.797 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:39.818 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:39.818 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:39.839 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:39.839 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:08:39.843 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:08:39.880 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:39.880 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:39.881 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:39.879+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:39.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:39.971 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:39.969+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:40.033 INFO:tasks.rados.rados.0.smithi016.stdout:225: expect (ObjNum 71 snap 3 seq_num 71) 2024-09-16T04:08:40.037 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:40.037 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:40.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:40.072 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:40.070+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:40.081 INFO:tasks.rados.rados.0.smithi016.stdout:225: done (6 left) 2024-09-16T04:08:40.081 INFO:tasks.rados.rados.0.smithi016.stdout:226: done (5 left) 2024-09-16T04:08:40.081 INFO:tasks.rados.rados.0.smithi016.stdout:228: done (4 left) 2024-09-16T04:08:40.081 INFO:tasks.rados.rados.0.smithi016.stdout:229: done (3 left) 2024-09-16T04:08:40.081 INFO:tasks.rados.rados.0.smithi016.stdout:230: done (2 left) 2024-09-16T04:08:40.081 INFO:tasks.rados.rados.0.smithi016.stdout:232: done (1 left) 2024-09-16T04:08:40.081 INFO:tasks.rados.rados.0.smithi016.stdout:233: copy_from oid 11 from oid 22 current snap is 18 2024-09-16T04:08:40.081 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 2 current snap is 18 2024-09-16T04:08:40.081 INFO:tasks.rados.rados.0.smithi016.stdout:234: seq_num 108 ranges {32768=655360,688128=606208,1294336=475136,1769472=81920} 2024-09-16T04:08:40.084 INFO:tasks.rados.rados.0.smithi016.stdout:234: writing smithi01639990-2 from 32768 to 688128 tid 1 2024-09-16T04:08:40.088 INFO:tasks.rados.rados.0.smithi016.stdout:234: writing smithi01639990-2 from 688128 to 1294336 tid 2 2024-09-16T04:08:40.091 INFO:tasks.rados.rados.0.smithi016.stdout:234: writing smithi01639990-2 from 1294336 to 1769472 tid 3 2024-09-16T04:08:40.091 INFO:tasks.rados.rados.0.smithi016.stdout:234: writing smithi01639990-2 from 1769472 to 1851392 tid 4 2024-09-16T04:08:40.091 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 44 current snap is 18 2024-09-16T04:08:40.091 INFO:tasks.rados.rados.0.smithi016.stdout:235: seq_num 109 ranges {0=671744,671744=753664,1425408=409600} 2024-09-16T04:08:40.096 INFO:tasks.rados.rados.0.smithi016.stdout:235: writing smithi01639990-44 from 0 to 671744 tid 1 2024-09-16T04:08:40.101 INFO:tasks.rados.rados.0.smithi016.stdout:235: writing smithi01639990-44 from 671744 to 1425408 tid 2 2024-09-16T04:08:40.104 INFO:tasks.rados.rados.0.smithi016.stdout:235: writing smithi01639990-44 from 1425408 to 1835008 tid 3 2024-09-16T04:08:40.104 INFO:tasks.rados.rados.0.smithi016.stdout:236: read oid 16 snap -1 2024-09-16T04:08:40.104 INFO:tasks.rados.rados.0.smithi016.stdout:236: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:237: delete oid 38 current snap is 18 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:233: finishing copy_from to smithi01639990-11 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:233: got expected ENOENT (src dne) 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:233: finishing copy_from racing read to smithi01639990-11 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:227: finishing write tid 5 to smithi01639990-50 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:227: finishing write tid 6 to smithi01639990-50 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:227: oid 50 updating version 0 to 43 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:227: oid 50 version 43 is already newer than 41 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:227: oid 50 version 43 is already newer than 40 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:227: oid 50 version 43 is already newer than 42 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:227: oid 50 updating version 43 to 44 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 50 v 44 (ObjNum 106 snap 18 seq_num 106) dirty exists 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:227: left oid 50 (ObjNum 106 snap 18 seq_num 106) 2024-09-16T04:08:40.105 INFO:tasks.rados.rados.0.smithi016.stdout:234: finishing write tid 1 to smithi01639990-2 2024-09-16T04:08:40.106 INFO:tasks.rados.rados.0.smithi016.stdout:234: finishing write tid 2 to smithi01639990-2 2024-09-16T04:08:40.106 INFO:tasks.rados.rados.0.smithi016.stdout:234: finishing write tid 3 to smithi01639990-2 2024-09-16T04:08:40.106 INFO:tasks.rados.rados.0.smithi016.stdout:234: finishing write tid 4 to smithi01639990-2 2024-09-16T04:08:40.106 INFO:tasks.rados.rados.0.smithi016.stdout:235: finishing write tid 1 to smithi01639990-44 2024-09-16T04:08:40.108 INFO:tasks.rados.rados.0.smithi016.stdout:235: finishing write tid 2 to smithi01639990-44 2024-09-16T04:08:40.108 INFO:tasks.rados.rados.0.smithi016.stdout:227: done (5 left) 2024-09-16T04:08:40.108 INFO:tasks.rados.rados.0.smithi016.stdout:233: done (4 left) 2024-09-16T04:08:40.109 INFO:tasks.rados.rados.0.smithi016.stdout:237: done (3 left) 2024-09-16T04:08:40.109 INFO:tasks.rados.rados.0.smithi016.stdout:238: copy_from oid 48 from oid 8 current snap is 18 2024-09-16T04:08:40.109 INFO:tasks.rados.rados.0.smithi016.stdout:239: read oid 30 snap -1 2024-09-16T04:08:40.109 INFO:tasks.rados.rados.0.smithi016.stdout:239: expect (ObjNum 103 snap 17 seq_num 103) 2024-09-16T04:08:40.109 INFO:tasks.rados.rados.0.smithi016.stdout:240: copy_from oid 38 from oid 47 current snap is 18 2024-09-16T04:08:40.109 INFO:tasks.rados.rados.0.smithi016.stdout:241: snap_create 2024-09-16T04:08:40.110 INFO:tasks.rados.rados.0.smithi016.stdout:235: finishing write tid 3 to smithi01639990-44 2024-09-16T04:08:40.110 INFO:tasks.rados.rados.0.smithi016.stdout:235: finishing write tid 4 to smithi01639990-44 2024-09-16T04:08:40.116 INFO:tasks.rados.rados.0.smithi016.stdout:235: finishing write tid 5 to smithi01639990-44 2024-09-16T04:08:40.116 INFO:tasks.rados.rados.0.smithi016.stdout:235: oid 44 updating version 0 to 54 2024-09-16T04:08:40.116 INFO:tasks.rados.rados.0.smithi016.stdout:235: oid 44 version 54 is already newer than 53 2024-09-16T04:08:40.116 INFO:tasks.rados.rados.0.smithi016.stdout:235: oid 44 version 54 is already newer than 51 2024-09-16T04:08:40.116 INFO:tasks.rados.rados.0.smithi016.stdout:235: oid 44 version 54 is already newer than 52 2024-09-16T04:08:40.116 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 44 v 54 (ObjNum 109 snap 18 seq_num 109) dirty exists 2024-09-16T04:08:40.116 INFO:tasks.rados.rados.0.smithi016.stdout:235: left oid 44 (ObjNum 109 snap 18 seq_num 109) 2024-09-16T04:08:40.130 INFO:tasks.rados.rados.0.smithi016.stdout:238: finishing copy_from racing read to smithi01639990-48 2024-09-16T04:08:40.132 INFO:tasks.rados.rados.0.smithi016.stdout:236: expect (ObjNum 57 snap 2 seq_num 57) 2024-09-16T04:08:40.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /var/log/ceph/exp.3.es2.1 2024-09-16T04:08:40.165 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:40.165 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:40.173 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:40.170+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:40.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:40.190 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.1 2024-09-16T04:08:40.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-1 show-label 2024-09-16T04:08:40.210 INFO:tasks.rados.rados.0.smithi016.stdout:238: finishing copy_from to smithi01639990-48 2024-09-16T04:08:40.210 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 48 v 29 (ObjNum 100 snap 17 seq_num 100) dirty exists 2024-09-16T04:08:40.210 INFO:tasks.rados.rados.0.smithi016.stdout:239: expect (ObjNum 103 snap 17 seq_num 103) 2024-09-16T04:08:40.252 INFO:teuthology.orchestra.run.smithi016.stdout:inferring bluefs devices from bluestore path 2024-09-16T04:08:40.293 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:40.291+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:40.319 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:40.319 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:40.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:40.366 INFO:tasks.rados.rados.0.smithi016.stdout:240: finishing copy_from racing read to smithi01639990-38 2024-09-16T04:08:40.366 INFO:tasks.rados.rados.0.smithi016.stdout:240: finishing copy_from to smithi01639990-38 2024-09-16T04:08:40.366 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 38 v 47 (ObjNum 94 snap 12 seq_num 94) dirty exists 2024-09-16T04:08:40.400 INFO:tasks.rados.rados.0.smithi016.stdout:234: finishing write tid 5 to smithi01639990-2 2024-09-16T04:08:40.401 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:40.399+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:40.403 INFO:tasks.rados.rados.0.smithi016.stdout:234: finishing write tid 6 to smithi01639990-2 2024-09-16T04:08:40.403 INFO:tasks.rados.rados.0.smithi016.stdout:234: oid 2 updating version 0 to 22 2024-09-16T04:08:40.403 INFO:tasks.rados.rados.0.smithi016.stdout:234: oid 2 version 22 is already newer than 20 2024-09-16T04:08:40.403 INFO:tasks.rados.rados.0.smithi016.stdout:234: oid 2 version 22 is already newer than 18 2024-09-16T04:08:40.403 INFO:tasks.rados.rados.0.smithi016.stdout:234: oid 2 version 22 is already newer than 19 2024-09-16T04:08:40.403 INFO:tasks.rados.rados.0.smithi016.stdout:234: oid 2 version 22 is already newer than 21 2024-09-16T04:08:40.403 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 2 v 22 (ObjNum 108 snap 18 seq_num 108) dirty exists 2024-09-16T04:08:40.403 INFO:tasks.rados.rados.0.smithi016.stdout:234: left oid 2 (ObjNum 108 snap 18 seq_num 108) 2024-09-16T04:08:40.434 INFO:tasks.rados.rados.0.smithi016.stdout:234: done (6 left) 2024-09-16T04:08:40.434 INFO:tasks.rados.rados.0.smithi016.stdout:235: done (5 left) 2024-09-16T04:08:40.434 INFO:tasks.rados.rados.0.smithi016.stdout:236: done (4 left) 2024-09-16T04:08:40.434 INFO:tasks.rados.rados.0.smithi016.stdout:238: done (3 left) 2024-09-16T04:08:40.434 INFO:tasks.rados.rados.0.smithi016.stdout:239: done (2 left) 2024-09-16T04:08:40.434 INFO:tasks.rados.rados.0.smithi016.stdout:240: done (1 left) 2024-09-16T04:08:40.434 INFO:tasks.rados.rados.0.smithi016.stdout:241: done (0 left) 2024-09-16T04:08:40.434 INFO:tasks.rados.rados.0.smithi016.stdout:242: delete oid 5 current snap is 19 2024-09-16T04:08:40.463 INFO:tasks.rados.rados.0.smithi016.stdout:242: done (0 left) 2024-09-16T04:08:40.463 INFO:tasks.rados.rados.0.smithi016.stdout:243: read oid 12 snap -1 2024-09-16T04:08:40.464 INFO:tasks.rados.rados.0.smithi016.stdout:243: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:08:40.464 INFO:tasks.rados.rados.0.smithi016.stdout:244: delete oid 7 current snap is 19 2024-09-16T04:08:40.467 INFO:tasks.rados.rados.0.smithi016.stdout:243: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:08:40.475 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:40.473+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:40.479 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:40.479 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:40.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:40.521 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:08:40.521 INFO:teuthology.orchestra.run.smithi016.stdout: "/var/lib/ceph/osd/ceph-1/block": { 2024-09-16T04:08:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_uuid": "8dc0488e-efb6-4664-a799-5e14c56769b0", 2024-09-16T04:08:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 96636764160, 2024-09-16T04:08:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "btime": "2024-09-16T04:06:47.544544+0000", 2024-09-16T04:08:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "main", 2024-09-16T04:08:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "bfm_blocks": "23592960", 2024-09-16T04:08:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "bfm_blocks_per_key": "128", 2024-09-16T04:08:40.522 INFO:teuthology.orchestra.run.smithi016.stdout: "bfm_bytes_per_block": "4096", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "bfm_size": "96636764160", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs": "1", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_fsid": "44381953-4509-4b15-853e-3887928dfa29", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version_when_created": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "created_at": "2024-09-16T04:06:50.181547Z", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "elastic_shared_blobs": "1", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": "16", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "kv_backend": "rocksdb", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "magic": "ceph osd volume v026", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "multi": "yes", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "ready": "ready", 2024-09-16T04:08:40.523 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "19", 2024-09-16T04:08:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bluestore", 2024-09-16T04:08:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": "1" 2024-09-16T04:08:40.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:08:40.524 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:08:40.525 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.1 2024-09-16T04:08:40.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-1 show-sharding 2024-09-16T04:08:40.612 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:40.610+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:40.632 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:40.633 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:40.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:40.677 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:40.675+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:40.778 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:40.776+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:40.788 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:40.789 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:40.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:40.879 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:40.877+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:40.948 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:40.948 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:40.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:40.981 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:40.979+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:41.065 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:41.065 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:41.073 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:41.081 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:41.079+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:41.185 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:41.183+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:41.189 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:41.190 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:41.199 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:41.214 INFO:tasks.rados.rados.0.smithi016.stdout:243: done (1 left) 2024-09-16T04:08:41.214 INFO:tasks.rados.rados.0.smithi016.stdout:244: done (0 left) 2024-09-16T04:08:41.214 INFO:tasks.rados.rados.0.smithi016.stdout:245: rollback oid 5 current snap is 19 2024-09-16T04:08:41.214 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 5 to 18 2024-09-16T04:08:41.214 INFO:tasks.rados.rados.0.smithi016.stdout:246: copy_from oid 41 from oid 32 current snap is 19 2024-09-16T04:08:41.214 INFO:tasks.rados.rados.0.smithi016.stdout:247: delete oid 39 current snap is 19 2024-09-16T04:08:41.219 INFO:tasks.rados.rados.0.smithi016.stdout:245: finishing rollback tid 1 to smithi01639990-5 2024-09-16T04:08:41.219 INFO:tasks.rados.rados.0.smithi016.stdout:247: done (2 left) 2024-09-16T04:08:41.219 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 19 current snap is 19 2024-09-16T04:08:41.219 INFO:tasks.rados.rados.0.smithi016.stdout:248: seq_num 114 ranges {1277952=753664,2031616=688128,2719744=475136} 2024-09-16T04:08:41.230 INFO:tasks.rados.rados.0.smithi016.stdout:248: writing smithi01639990-19 from 1277952 to 2031616 tid 1 2024-09-16T04:08:41.234 INFO:tasks.rados.rados.0.smithi016.stdout:248: writing smithi01639990-19 from 2031616 to 2719744 tid 2 2024-09-16T04:08:41.237 INFO:tasks.rados.rados.0.smithi016.stdout:248: writing smithi01639990-19 from 2719744 to 3194880 tid 3 2024-09-16T04:08:41.237 INFO:tasks.rados.rados.0.smithi016.stdout:249: rmattr oid 8 current snap is 19 2024-09-16T04:08:41.237 INFO:tasks.rados.rados.0.smithi016.stdout:249: done (3 left) 2024-09-16T04:08:41.237 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 37 current snap is 19 2024-09-16T04:08:41.237 INFO:tasks.rados.rados.0.smithi016.stdout:245: finishing rollback tid 2 to smithi01639990-5 2024-09-16T04:08:41.238 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 5 v 20 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:08:41.238 INFO:tasks.rados.rados.0.smithi016.stdout:246: finishing copy_from to smithi01639990-41 2024-09-16T04:08:41.238 INFO:tasks.rados.rados.0.smithi016.stdout:246: got expected ENOENT (src dne) 2024-09-16T04:08:41.238 INFO:tasks.rados.rados.0.smithi016.stdout:246: finishing copy_from racing read to smithi01639990-41 2024-09-16T04:08:41.238 INFO:tasks.rados.rados.0.smithi016.stdout:250: seq_num 115 ranges {2211840=770048,2981888=704512,3686400=458752} 2024-09-16T04:08:41.253 INFO:tasks.rados.rados.0.smithi016.stdout:250: writing smithi01639990-37 from 2211840 to 2981888 tid 1 2024-09-16T04:08:41.258 INFO:tasks.rados.rados.0.smithi016.stdout:250: writing smithi01639990-37 from 2981888 to 3686400 tid 2 2024-09-16T04:08:41.262 INFO:tasks.rados.rados.0.smithi016.stdout:250: writing smithi01639990-37 from 3686400 to 4145152 tid 3 2024-09-16T04:08:41.262 INFO:tasks.rados.rados.0.smithi016.stdout:245: done (3 left) 2024-09-16T04:08:41.262 INFO:tasks.rados.rados.0.smithi016.stdout:246: done (2 left) 2024-09-16T04:08:41.262 INFO:tasks.rados.rados.0.smithi016.stdout:251: rollback oid 46 current snap is 19 2024-09-16T04:08:41.262 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 46 to 18 2024-09-16T04:08:41.262 INFO:tasks.rados.rados.0.smithi016.stdout:248: finishing write tid 1 to smithi01639990-19 2024-09-16T04:08:41.262 INFO:tasks.rados.rados.0.smithi016.stdout:248: finishing write tid 2 to smithi01639990-19 2024-09-16T04:08:41.262 INFO:tasks.rados.rados.0.smithi016.stdout:248: finishing write tid 3 to smithi01639990-19 2024-09-16T04:08:41.262 INFO:tasks.rados.rados.0.smithi016.stdout:252: delete oid 42 current snap is 19 2024-09-16T04:08:41.263 INFO:tasks.rados.rados.0.smithi016.stdout:248: finishing write tid 4 to smithi01639990-19 2024-09-16T04:08:41.263 INFO:tasks.rados.rados.0.smithi016.stdout:248: finishing write tid 5 to smithi01639990-19 2024-09-16T04:08:41.263 INFO:tasks.rados.rados.0.smithi016.stdout:248: oid 19 updating version 0 to 20 2024-09-16T04:08:41.263 INFO:tasks.rados.rados.0.smithi016.stdout:248: oid 19 updating version 20 to 22 2024-09-16T04:08:41.263 INFO:tasks.rados.rados.0.smithi016.stdout:248: oid 19 version 22 is already newer than 21 2024-09-16T04:08:41.263 INFO:tasks.rados.rados.0.smithi016.stdout:248: oid 19 version 22 is already newer than 19 2024-09-16T04:08:41.263 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 19 v 22 (ObjNum 114 snap 19 seq_num 114) dirty exists 2024-09-16T04:08:41.263 INFO:tasks.rados.rados.0.smithi016.stdout:248: left oid 19 (ObjNum 114 snap 19 seq_num 114) 2024-09-16T04:08:41.263 INFO:tasks.rados.rados.0.smithi016.stdout:250: finishing write tid 1 to smithi01639990-37 2024-09-16T04:08:41.264 INFO:tasks.rados.rados.0.smithi016.stdout:250: finishing write tid 2 to smithi01639990-37 2024-09-16T04:08:41.266 INFO:tasks.rados.rados.0.smithi016.stdout:248: done (3 left) 2024-09-16T04:08:41.266 INFO:tasks.rados.rados.0.smithi016.stdout:252: done (2 left) 2024-09-16T04:08:41.266 INFO:tasks.rados.rados.0.smithi016.stdout:253: read oid 13 snap -1 2024-09-16T04:08:41.266 INFO:tasks.rados.rados.0.smithi016.stdout:253: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-16T04:08:41.266 INFO:tasks.rados.rados.0.smithi016.stdout:254: copy_from oid 50 from oid 11 current snap is 19 2024-09-16T04:08:41.267 INFO:tasks.rados.rados.0.smithi016.stdout:255: read oid 30 snap -1 2024-09-16T04:08:41.267 INFO:tasks.rados.rados.0.smithi016.stdout:255: expect (ObjNum 103 snap 17 seq_num 103) 2024-09-16T04:08:41.267 INFO:tasks.rados.rados.0.smithi016.stdout:256: rollback oid 16 current snap is 19 2024-09-16T04:08:41.267 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 16 to 14 2024-09-16T04:08:41.267 INFO:tasks.rados.rados.0.smithi016.stdout:257: snap_create 2024-09-16T04:08:41.270 INFO:tasks.rados.rados.0.smithi016.stdout:250: finishing write tid 3 to smithi01639990-37 2024-09-16T04:08:41.271 INFO:tasks.rados.rados.0.smithi016.stdout:253: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-16T04:08:41.278 INFO:tasks.rados.rados.0.smithi016.stdout:250: finishing write tid 4 to smithi01639990-37 2024-09-16T04:08:41.278 INFO:tasks.rados.rados.0.smithi016.stdout:251: finishing rollback tid 0 to smithi01639990-46 2024-09-16T04:08:41.278 INFO:tasks.rados.rados.0.smithi016.stdout:251: finishing rollback tid 1 to smithi01639990-46 2024-09-16T04:08:41.278 INFO:tasks.rados.rados.0.smithi016.stdout:256: finishing rollback tid 0 to smithi01639990-16 2024-09-16T04:08:41.278 INFO:tasks.rados.rados.0.smithi016.stdout:250: finishing write tid 5 to smithi01639990-37 2024-09-16T04:08:41.278 INFO:tasks.rados.rados.0.smithi016.stdout:250: oid 37 updating version 0 to 81 2024-09-16T04:08:41.278 INFO:tasks.rados.rados.0.smithi016.stdout:250: oid 37 updating version 81 to 84 2024-09-16T04:08:41.279 INFO:tasks.rados.rados.0.smithi016.stdout:250: oid 37 version 84 is already newer than 83 2024-09-16T04:08:41.279 INFO:tasks.rados.rados.0.smithi016.stdout:250: oid 37 version 84 is already newer than 82 2024-09-16T04:08:41.279 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 37 v 84 (ObjNum 115 snap 19 seq_num 115) dirty exists 2024-09-16T04:08:41.279 INFO:tasks.rados.rados.0.smithi016.stdout:250: left oid 37 (ObjNum 115 snap 19 seq_num 115) 2024-09-16T04:08:41.279 INFO:tasks.rados.rados.0.smithi016.stdout:251: finishing rollback tid 2 to smithi01639990-46 2024-09-16T04:08:41.279 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 46 v 78 (ObjNum 82 snap 7 seq_num 82) dirty exists 2024-09-16T04:08:41.279 INFO:tasks.rados.rados.0.smithi016.stdout:256: finishing rollback tid 1 to smithi01639990-16 2024-09-16T04:08:41.287 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:41.285+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:41.288 INFO:tasks.rados.rados.0.smithi016.stdout:254: finishing copy_from racing read to smithi01639990-50 2024-09-16T04:08:41.290 INFO:tasks.rados.rados.0.smithi016.stdout:254: finishing copy_from to smithi01639990-50 2024-09-16T04:08:41.290 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 50 v 41 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-09-16T04:08:41.290 INFO:tasks.rados.rados.0.smithi016.stdout:255: expect (ObjNum 103 snap 17 seq_num 103) 2024-09-16T04:08:41.354 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:41.354 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:41.361 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:41.390 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:41.388+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:41.464 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:41.464 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:41.473 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:41.491 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:41.489+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:41.518 INFO:tasks.rados.rados.0.smithi016.stdout:250: done (6 left) 2024-09-16T04:08:41.519 INFO:tasks.rados.rados.0.smithi016.stdout:251: done (5 left) 2024-09-16T04:08:41.519 INFO:tasks.rados.rados.0.smithi016.stdout:253: done (4 left) 2024-09-16T04:08:41.519 INFO:tasks.rados.rados.0.smithi016.stdout:254: done (3 left) 2024-09-16T04:08:41.519 INFO:tasks.rados.rados.0.smithi016.stdout:255: done (2 left) 2024-09-16T04:08:41.519 INFO:tasks.rados.rados.0.smithi016.stdout:257: done (1 left) 2024-09-16T04:08:41.519 INFO:tasks.rados.rados.0.smithi016.stdout:258: setattr oid 31 current snap is 20 2024-09-16T04:08:41.519 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 18 current snap is 20 2024-09-16T04:08:41.519 INFO:tasks.rados.rados.0.smithi016.stdout:259: seq_num 117 ranges {0=425984,425984=606208,1032192=540672,1572864=409600} 2024-09-16T04:08:41.521 INFO:tasks.rados.rados.0.smithi016.stdout:259: writing smithi01639990-18 from 0 to 425984 tid 1 2024-09-16T04:08:41.525 INFO:tasks.rados.rados.0.smithi016.stdout:259: writing smithi01639990-18 from 425984 to 1032192 tid 2 2024-09-16T04:08:41.529 INFO:tasks.rados.rados.0.smithi016.stdout:259: writing smithi01639990-18 from 1032192 to 1572864 tid 3 2024-09-16T04:08:41.532 INFO:tasks.rados.rados.0.smithi016.stdout:259: writing smithi01639990-18 from 1572864 to 1982464 tid 4 2024-09-16T04:08:41.532 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 9 current snap is 20 2024-09-16T04:08:41.532 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 31 v 18 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T04:08:41.532 INFO:tasks.rados.rados.0.smithi016.stdout:260: seq_num 118 ranges {1851392=442368,2293760=753664,3047424=655360,3702784=147456} 2024-09-16T04:08:41.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:08:41.544 INFO:tasks.rados.rados.0.smithi016.stdout:260: writing smithi01639990-9 from 1851392 to 2293760 tid 1 2024-09-16T04:08:41.549 INFO:tasks.rados.rados.0.smithi016.stdout:260: writing smithi01639990-9 from 2293760 to 3047424 tid 2 2024-09-16T04:08:41.553 INFO:tasks.rados.rados.0.smithi016.stdout:260: writing smithi01639990-9 from 3047424 to 3702784 tid 3 2024-09-16T04:08:41.554 INFO:tasks.rados.rados.0.smithi016.stdout:260: writing smithi01639990-9 from 3702784 to 3850240 tid 4 2024-09-16T04:08:41.555 INFO:tasks.rados.rados.0.smithi016.stdout:258: done (3 left) 2024-09-16T04:08:41.555 INFO:tasks.rados.rados.0.smithi016.stdout:261: read oid 37 snap -1 2024-09-16T04:08:41.555 INFO:tasks.rados.rados.0.smithi016.stdout:261: expect (ObjNum 115 snap 19 seq_num 115) 2024-09-16T04:08:41.555 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 22 current snap is 20 2024-09-16T04:08:41.555 INFO:tasks.rados.rados.0.smithi016.stdout:259: finishing write tid 1 to smithi01639990-18 2024-09-16T04:08:41.555 INFO:tasks.rados.rados.0.smithi016.stdout:259: finishing write tid 2 to smithi01639990-18 2024-09-16T04:08:41.555 INFO:tasks.rados.rados.0.smithi016.stdout:262: seq_num 119 ranges {0=458752,458752=507904,966656=507904,1474560=458752,1933312=81920} 2024-09-16T04:08:41.557 INFO:tasks.rados.rados.0.smithi016.stdout:262: writing smithi01639990-22 from 0 to 458752 tid 1 2024-09-16T04:08:41.561 INFO:tasks.rados.rados.0.smithi016.stdout:262: writing smithi01639990-22 from 458752 to 966656 tid 2 2024-09-16T04:08:41.564 INFO:tasks.rados.rados.0.smithi016.stdout:262: writing smithi01639990-22 from 966656 to 1474560 tid 3 2024-09-16T04:08:41.567 INFO:tasks.rados.rados.0.smithi016.stdout:262: writing smithi01639990-22 from 1474560 to 1933312 tid 4 2024-09-16T04:08:41.568 INFO:tasks.rados.rados.0.smithi016.stdout:262: writing smithi01639990-22 from 1933312 to 2015232 tid 5 2024-09-16T04:08:41.568 INFO:tasks.rados.rados.0.smithi016.stdout:263: copy_from oid 41 from oid 39 current snap is 20 2024-09-16T04:08:41.568 INFO:tasks.rados.rados.0.smithi016.stdout:264: rollback oid 23 current snap is 20 2024-09-16T04:08:41.568 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 23 to 19 2024-09-16T04:08:41.568 INFO:tasks.rados.rados.0.smithi016.stdout:265: snap_remove snap 18 2024-09-16T04:08:41.593 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:41.594 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:41.594 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:41.591+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:41.600 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:41.656 INFO:teuthology.orchestra.run.smithi016.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-09-16T04:08:41.660 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.1 2024-09-16T04:08:41.660 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-16T04:08:41.661 INFO:tasks.thrashosds.thrasher:applying new sharding: O(1,8-16) P(1,8-38) 2024-09-16T04:08:41.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-1 --sharding 'O(1,8-16) P(1,8-38)' reshard 2024-09-16T04:08:41.702 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:41.700+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:41.743 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:41.744 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:41.752 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:41.795 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:41.793+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:41.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:41.857 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:41.857 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:41.866 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:41.910 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:41.908+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:41.971 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:41.971 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:41.976 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:41.976 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:41.976 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T04:08:41.979 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:41.997 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:41.995+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:41.998 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:42.018 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:42.042 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:42.042 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:42.063 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:42.084 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:42.098 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:42.096+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:42.105 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:42.106 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:42.123 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:42.123 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:42.123 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:42.123 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:42.124 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:42.127 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:42.127 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:42.132 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:42.149 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:42.149 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:42.170 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:42.170 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:42.191 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:08:42.195 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:08:42.199 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:42.196+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:42.236 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s3 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.13039921, 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0068480019999999997, 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.237 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103703+0000", 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103704+0000", 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7100000000000001e-07 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103727+0000", 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3694999999999999e-05 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103810+0000", 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3091999999999997e-05 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103816+0000", 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4519999999999996e-06 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103971+0000", 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000154993 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103982+0000", 2024-09-16T04:08:42.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0933e-05 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110491+0000", 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065096800000000003 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110551+0000", 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9985999999999998e-05 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 33/27 ECSubWrite(tid=14, reqid=osd.4.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=33'10))", 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.130349871999996, 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067325809999999996, 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103753+0000", 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103754+0000", 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5899999999999995e-07 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103768+0000", 2024-09-16T04:08:42.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4562999999999999e-05 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103805+0000", 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6730000000000002e-05 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103813+0000", 2024-09-16T04:08:42.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5499999999999995e-06 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103868+0000", 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4475000000000003e-05 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103877+0000", 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7099999999999996e-06 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110444+0000", 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065675170000000001 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110485+0000", 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1177000000000002e-05 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 37/27 ECSubWrite(tid=124, reqid=osd.3.0:124, at_version=37'22, trim_to=0'0, roll_forward_to=35'20))", 2024-09-16T04:08:42.246 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.130765572000001, 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0070925160000000001, 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103337+0000", 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6999999999999999e-07 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103359+0000", 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1835999999999999e-05 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103683+0000", 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00032427399999999999 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103693+0000", 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7380000000000006e-06 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103756+0000", 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3115999999999997e-05 2024-09-16T04:08:42.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.103766+0000", 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0043e-05 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110407+0000", 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0066407230000000003 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.110430+0000", 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2616000000000002e-05 2024-09-16T04:08:42.250 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 37/27 ECSubWrite(tid=72, reqid=osd.5.0:72, at_version=37'22, trim_to=0'0, roll_forward_to=35'18))", 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.129935299, 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.008202661, 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104167+0000", 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104168+0000", 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0200000000000001e-07 2024-09-16T04:08:42.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104188+0000", 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9933999999999999e-05 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104232+0000", 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4196000000000002e-05 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104241+0000", 2024-09-16T04:08:42.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7979999999999995e-06 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104928+0000", 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000687124 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.104937+0000", 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5269999999999998e-06 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112328+0000", 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0073914510000000003 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.112370+0000", 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1928999999999998e-05 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.3s3 37/27 ECSubWrite(tid=22, reqid=osd.4.0:22, at_version=37'16, trim_to=0'0, roll_forward_to=35'14))", 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:42.255 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.127215691, 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082659529999999995, 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106887+0000", 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1600000000000002e-07 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106910+0000", 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2954000000000001e-05 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106990+0000", 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9788000000000002e-05 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.106997+0000", 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4379999999999998e-06 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107267+0000", 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026999599999999999 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.107276+0000", 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9530000000000005e-06 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115126+0000", 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0078491940000000003 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:13.115153+0000", 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7214000000000001e-05 2024-09-16T04:08:42.259 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.784162466000002, 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091491550000000008, 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5800000000000001e-07 2024-09-16T04:08:42.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450512+0000", 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057160000000000002 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450541+0000", 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8835000000000001e-05 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450549+0000", 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3080000000000008e-06 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450607+0000", 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7763999999999998e-05 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450616+0000", 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459045+0000", 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084290709999999998 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459089+0000", 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4233000000000002e-05 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.264 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=160, reqid=client.4440.0:347, at_version=42'32, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.781540383999999, 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065520760000000004, 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452562+0000", 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452563+0000", 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6300000000000005e-07 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452621+0000", 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8488e-05 2024-09-16T04:08:42.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452654+0000", 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2750999999999999e-05 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452659+0000", 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.293e-06 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452730+0000", 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0173999999999996e-05 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452738+0000", 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6260000000000001e-06 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459095+0000", 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0063566530000000003 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459114+0000", 2024-09-16T04:08:42.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9428000000000001e-05 2024-09-16T04:08:42.269 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.269 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.269 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.270 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=161, reqid=client.4440.0:348, at_version=42'33, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:42.270 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:42.270 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.781477777999999, 2024-09-16T04:08:42.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065540390000000002, 2024-09-16T04:08:42.270 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.271 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.271 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.271 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:42.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:42.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452625+0000", 2024-09-16T04:08:42.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6300000000000001e-07 2024-09-16T04:08:42.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452632+0000", 2024-09-16T04:08:42.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6699999999999997e-06 2024-09-16T04:08:42.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452678+0000", 2024-09-16T04:08:42.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5760000000000002e-05 2024-09-16T04:08:42.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.452682+0000", 2024-09-16T04:08:42.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.036e-06 2024-09-16T04:08:42.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453280+0000", 2024-09-16T04:08:42.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059861699999999996 2024-09-16T04:08:42.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.453289+0000", 2024-09-16T04:08:42.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9590000000000001e-06 2024-09-16T04:08:42.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459117+0000", 2024-09-16T04:08:42.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0058281770000000004 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459179+0000", 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1556999999999994e-05 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:469 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~491520] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:08:42.278 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.681329502000001, 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098323030000000006, 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 469 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:08:42.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552774+0000", 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1999999999999999e-06 2024-09-16T04:08:42.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553284+0000", 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00050997900000000003 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553297+0000", 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2723e-05 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553306+0000", 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5969999999999992e-06 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553363+0000", 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6922999999999997e-05 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553436+0000", 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3090000000000007e-05 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.554087+0000", 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065173499999999997 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.558220+0000", 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041326330000000001 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562517+0000", 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042965859999999998 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562605+0000", 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8837000000000003e-05 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:470 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~589824] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:42.284 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.556072+0000", 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.67803097, 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079188939999999992, 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 470 2024-09-16T04:08:42.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556072+0000", 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556072+0000", 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.286 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556073+0000", 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0729999999999999e-06 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556674+0000", 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00060172999999999997 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556685+0000", 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0111e-05 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556689+0000", 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8420000000000004e-06 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556732+0000", 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2654000000000002e-05 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.556832+0000", 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9953999999999999e-05 2024-09-16T04:08:42.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.557727+0000", 2024-09-16T04:08:42.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089499499999999999 2024-09-16T04:08:42.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562616+0000", 2024-09-16T04:08:42.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0048893870000000002 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563944+0000", 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013275380000000001 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563991+0000", 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6610000000000003e-05 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.290 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:471 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~557056] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.559406+0000", 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.674696407999999, 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0071509360000000001, 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.291 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 471 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.559406+0000", 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.559406+0000", 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.559407+0000", 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2780000000000001e-06 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560034+0000", 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00062697999999999996 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560044+0000", 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560050+0000", 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1059999999999996e-06 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560093+0000", 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2673000000000003e-05 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560217+0000", 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124175 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.295 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.561108+0000", 2024-09-16T04:08:42.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00089074 2024-09-16T04:08:42.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566027+0000", 2024-09-16T04:08:42.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049195159999999996 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566499+0000", 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000472032 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566557+0000", 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7989999999999999e-05 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.297 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:472 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~16384] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.560062+0000", 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.674040596999999, 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067563500000000004, 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:42.298 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 472 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560062+0000", 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560062+0000", 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560062+0000", 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8299999999999997e-07 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560072+0000", 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8539999999999992e-06 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560076+0000", 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1459999999999998e-06 2024-09-16T04:08:42.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560080+0000", 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.168e-06 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.561729+0000", 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001649799 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.561798+0000", 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8555999999999999e-05 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562168+0000", 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037029100000000002 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566620+0000", 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.004451404 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566775+0000", 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015578599999999999 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566818+0000", 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2862999999999999e-05 2024-09-16T04:08:42.303 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:473 3.as0 3:5e52341f:::smithi01639990-38:head [setxattr _header (57) in=7b] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.560085+0000", 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.674017815999999, 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084263350000000001, 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.304 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 473 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560085+0000", 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560085+0000", 2024-09-16T04:08:42.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560085+0000", 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8099999999999999e-07 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560087+0000", 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7880000000000001e-06 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560090+0000", 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3409999999999999e-06 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560093+0000", 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.672e-06 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562731+0000", 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0026385169999999999 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562783+0000", 2024-09-16T04:08:42.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1215999999999999e-05 2024-09-16T04:08:42.308 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563136+0000", 2024-09-16T04:08:42.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035292899999999998 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.566622+0000", 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.003486556 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568451+0000", 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0018289389999999999 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568511+0000", 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0096000000000002e-05 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:474 3.as0 3:5e52341f:::smithi01639990-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.673997541, 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097686969999999998, 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.310 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 474 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560106+0000", 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.63e-07 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560109+0000", 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.303e-06 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560116+0000", 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3860000000000001e-06 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560122+0000", 2024-09-16T04:08:42.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4450000000000004e-06 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563686+0000", 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035639740000000001 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563776+0000", 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0278000000000005e-05 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568854+0000", 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0050781680000000001 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568970+0000", 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011579899999999999 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.569874+0000", 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00090388100000000004 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.315 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.7963658460000005, 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015912599999999999, 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.316 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:08:42.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437757+0000", 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9735e-05 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437829+0000", 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1605000000000003e-05 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437838+0000", 2024-09-16T04:08:42.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442352+0000", 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045136799999999999 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442364+0000", 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2642e-05 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453618+0000", 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011254128 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453649+0000", 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1016000000000003e-05 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:08:42.320 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.7953272370000004, 2024-09-16T04:08:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014897297, 2024-09-16T04:08:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:08:42.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438776+0000", 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3099999999999997e-07 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438797+0000", 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0596000000000001e-05 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438836+0000", 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9684000000000002e-05 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438845+0000", 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6479999999999992e-06 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442677+0000", 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038317239999999999 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442685+0000", 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453654+0000", 2024-09-16T04:08:42.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010968637 2024-09-16T04:08:42.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453673+0000", 2024-09-16T04:08:42.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8967999999999999e-05 2024-09-16T04:08:42.325 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.793337288, 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012700487999999999, 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.326 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8900000000000001e-07 2024-09-16T04:08:42.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440785+0000", 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9850000000000001e-05 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445485+0000", 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046992329999999997 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445495+0000", 2024-09-16T04:08:42.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0506000000000001e-05 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445668+0000", 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000172905 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445679+0000", 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453438+0000", 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077588930000000002 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453466+0000", 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8175e-05 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=110, reqid=osd.5.0:109, at_version=0'0, trim_to=0'0, roll_forward_to=57'31))", 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:08:42.330 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.7695328359999998, 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011518758, 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:08:42.331 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8e-07 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464577+0000", 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9940000000000003e-06 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.332 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472531+0000", 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079544330000000003 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472538+0000", 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7789999999999998e-06 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472599+0000", 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0841999999999998e-05 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.333 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472610+0000", 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1185999999999999e-05 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476060+0000", 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034496650000000002 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476089+0000", 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8679000000000001e-05 2024-09-16T04:08:42.334 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=37, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.7460300340000003, 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.039596059000000003, 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.335 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1e-07 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.336 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488075+0000", 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8789999999999999e-06 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526262+0000", 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.038187556999999997 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526270+0000", 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3209999999999999e-06 2024-09-16T04:08:42.337 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526424+0000", 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015469 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.527669+0000", 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012444019999999999 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.338 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=38, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.7035377569999994, 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028484058, 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.339 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530566+0000", 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.85e-07 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530570+0000", 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7319999999999998e-06 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.340 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558866+0000", 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028295515 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558875+0000", 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7849999999999997e-06 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558966+0000", 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1611000000000001e-05 2024-09-16T04:08:42.341 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.559049+0000", 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.263e-05 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.342 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:42.343 INFO:teuthology.orchestra.run.smithi016.stdout:reshard success 2024-09-16T04:08:42.344 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:42.342+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:42.344 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:42.444 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:42.442+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:42.489 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:42.489 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:42.489 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:42.489 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:42.498 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:42.523 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-16T04:08:42.525 INFO:tasks.rados.rados.0.smithi016.stdout:265: done (7 left) 2024-09-16T04:08:42.525 INFO:tasks.rados.rados.0.smithi016.stdout:266: snap_remove snap 16 2024-09-16T04:08:42.582 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:42.580+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:42.644 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:42.644 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:42.644 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:42.644 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:42.644 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:42.646 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:42.644+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:42.652 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:42.747 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:42.745+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:42.785 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.1 2024-09-16T04:08:42.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-1 fsck 2024-09-16T04:08:42.798 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:42.798 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:42.798 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:42.798 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:42.798 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 52.854933719999998, 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.799 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:42.800 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.801 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:42.802 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:42.803 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.804 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.805 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 51.872529477, 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.806 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.807 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.808 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:42.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:42.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.811 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 51.862166971000001, 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.812 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.813 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.814 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.815 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 50.365578919000001, 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.816 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:42.817 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:42.817 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:42.817 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.817 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.817 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.820 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.820 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:42.821 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.821 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.821 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.821 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.822 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:42.823 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:42.824 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.825 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.826 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:42.826 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:42.826 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.826 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.826 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.826 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.826 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.826 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.911004425999998, 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.827 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:42.828 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:42.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:42.830 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:42.831 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:42.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.831 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:42.831 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:42.831 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:42.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.831 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.831 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:42.832 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.910950364999998, 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:42.833 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.836 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.837 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:42.838 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.686681827000001, 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.839 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.840 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:42.841 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.842 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.108747866999998, 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.843 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:42.846 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s2 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.695973637999998, 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010866819, 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099684+0000", 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.004e-06 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099714+0000", 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9807000000000001e-05 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099769+0000", 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5306000000000002e-05 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099781+0000", 2024-09-16T04:08:42.849 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2091e-05 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099835+0000", 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4058000000000003e-05 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099845+0000", 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5729999999999997e-06 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.850 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110509+0000", 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010664339 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110550+0000", 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640999999999999e-05 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:42.851 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.691934721999999, 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.852 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.853 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.854 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.855 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.690547307999999, 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.856 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.857 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.858 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.859 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=79, reqid=client.4440.0:208, at_version=33'22, trim_to=0'0, roll_forward_to=33'19))", 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.688280253000002, 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099172860000000009, 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.860 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107376+0000", 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.861 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107377+0000", 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8900000000000001e-07 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107437+0000", 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0628000000000003e-05 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107526+0000", 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8873000000000001e-05 2024-09-16T04:08:42.862 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107535+0000", 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0489999999999993e-06 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111543+0000", 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0040080810000000001 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.863 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.111554+0000", 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0321e-05 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117271+0000", 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0057176529999999996 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117294+0000", 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2391999999999998e-05 2024-09-16T04:08:42.864 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=188, reqid=client.4440.0:379, at_version=47'56, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.343415851, 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011758437, 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.865 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:42.866 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452266+0000", 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4151e-05 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459768+0000", 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075025179999999997 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.867 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459778+0000", 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0244999999999999e-05 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459835+0000", 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6394999999999998e-05 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459845+0000", 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0518e-05 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.868 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463963+0000", 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041182450000000004 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463999+0000", 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5667999999999998e-05 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.869 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.331228278000001, 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.870 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.871 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:08:42.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:08:42.873 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:426 3.13s0 3:ce079411:::smithi01639990-2:d [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.288155952, 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097940059999999992, 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:42.874 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 426 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:42.875 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507501+0000", 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4799999999999999e-07 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507505+0000", 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0149999999999997e-06 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.876 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507511+0000", 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4540000000000004e-06 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.507517+0000", 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0749999999999997e-06 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.508922+0000", 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001405211 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.877 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.509012+0000", 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9776000000000002e-05 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.517295+0000", 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0082831269999999995 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.878 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.358358328, 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019153804, 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.879 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437299+0000", 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437320+0000", 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1121999999999999e-05 2024-09-16T04:08:42.880 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437388+0000", 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7889e-05 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437397+0000", 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4390000000000006e-06 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441658+0000", 2024-09-16T04:08:42.881 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042607110000000004 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441667+0000", 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456416+0000", 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014748493 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.882 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456452+0000", 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6159999999999999e-05 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.355168617, 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017174697999999999, 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.883 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.884 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.885 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440489+0000", 2024-09-16T04:08:42.885 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.015e-06 2024-09-16T04:08:42.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.885 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.885 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440509+0000", 2024-09-16T04:08:42.885 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0338000000000001e-05 2024-09-16T04:08:42.885 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.885 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440547+0000", 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440557+0000", 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7820000000000005e-06 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.886 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442068+0000", 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015112179999999999 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442078+0000", 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4620000000000002e-06 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457639+0000", 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015560954 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.887 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457663+0000", 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4121000000000002e-05 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=82, reqid=osd.8.0:81, at_version=0'0, trim_to=0'0, roll_forward_to=57'51))", 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.335444297, 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010694719, 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.888 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.889 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460213+0000", 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4999999999999999e-07 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460222+0000", 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2669999999999995e-06 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460651+0000", 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00042901799999999998 2024-09-16T04:08:42.890 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.462020+0000", 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013696680000000001 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.466451+0000", 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044306529999999997 2024-09-16T04:08:42.891 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.466460+0000", 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.470859+0000", 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043990979999999997 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.470907+0000", 2024-09-16T04:08:42.892 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7679000000000002e-05 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s5 63/56 ECSubWrite(tid=277, reqid=osd.3.0:277, at_version=63'79, trim_to=0'0, roll_forward_to=62'77))", 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:38.511350+0000", 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.2843070030000003, 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0099420189999999999, 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.893 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511350+0000", 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511350+0000", 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511350+0000", 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8999999999999995e-07 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.894 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511375+0000", 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4448e-05 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511412+0000", 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7487000000000002e-05 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511422+0000", 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1900000000000001e-06 2024-09-16T04:08:42.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511486+0000", 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4579000000000006e-05 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511496+0000", 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0096e-05 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.521235+0000", 2024-09-16T04:08:42.896 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097389720000000003 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.521292+0000", 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6357000000000003e-05 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.897 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.1s7 63/56 ECSubWrite(tid=110, reqid=osd.0.0:110, at_version=63'19, trim_to=0'0, roll_forward_to=62'17))", 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:38.514015+0000", 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.281641757, 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010539229000000001, 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:42.898 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514015+0000", 2024-09-16T04:08:42.899 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.899 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.899 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.899 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:42.899 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514015+0000", 2024-09-16T04:08:42.899 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:42.899 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.899 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.899 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514015+0000", 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1400000000000001e-07 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514046+0000", 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1149000000000003e-05 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514148+0000", 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010135499999999999 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.900 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514159+0000", 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0961000000000001e-05 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514226+0000", 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7646999999999998e-05 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514237+0000", 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1042e-05 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.901 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.524512+0000", 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010274777000000001 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.524554+0000", 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2083999999999999e-05 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:42.902 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:42.904 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:42.905 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:42.901+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:43.004 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:43.001+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:43.009 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:43.009 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:43.009 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:43.009 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:43.018 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:43.106 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:43.104+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:43.121 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:43.121 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:43.121 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:43.121 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:43.121 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:43.130 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:43.206 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:43.204+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:43.235 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:43.235 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:43.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.090974838999998, 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:43.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:43.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:296 3.9s0 3:9fd10cc9:::smithi01639990-50:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:08:43.240 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.128094949000001, 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010626313, 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104575+0000", 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1349999999999999e-06 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104586+0000", 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0603e-05 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104606+0000", 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9840999999999999e-05 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104615+0000", 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0189999999999995e-06 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104674+0000", 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9534999999999998e-05 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104680+0000", 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9320000000000001e-06 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109134+0000", 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044534120000000003 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109195+0000", 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0983000000000002e-05 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109711+0000", 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051621600000000001 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.110753+0000", 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001042533 2024-09-16T04:08:43.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115115+0000", 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043613030000000004 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115200+0000", 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4801000000000003e-05 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.246 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.783645578000002, 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085197069999999996, 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:43.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449023+0000", 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4000000000000001e-07 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.449160+0000", 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013637999999999999 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451237+0000", 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020768179999999998 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451246+0000", 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451532+0000", 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00028571800000000002 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.451542+0000", 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0118999999999999e-05 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457500+0000", 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059576680000000002 2024-09-16T04:08:43.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457542+0000", 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2716999999999998e-05 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.772919230999999, 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.251 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:08:43.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.642377071999999, 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:43.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:08:43.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:08:43.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:363 3.9s0 3:94de6fe1:::smithi01639990-3:head [stat,delete] snapc 8=[6] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.792810412000001, 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085012569999999999, 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.260 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 363 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439858+0000", 2024-09-16T04:08:43.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439860+0000", 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9199999999999998e-06 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439869+0000", 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3170000000000008e-06 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439888+0000", 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9293e-05 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439897+0000", 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7669999999999992e-06 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439953+0000", 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6317999999999998e-05 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.439959+0000", 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3040000000000004e-06 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441596+0000", 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001637655 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.441659+0000", 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2658999999999998e-05 2024-09-16T04:08:43.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.442297+0000", 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00063811800000000002 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.443562+0000", 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001265361 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448304+0000", 2024-09-16T04:08:43.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047413560000000004 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.448359+0000", 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5189000000000003e-05 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.561454519000002, 2024-09-16T04:08:43.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:08:43.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.401050461000001, 2024-09-16T04:08:43.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:419 3.9s0 3:94de6fe1:::smithi01639990-3:head [append 0~786432] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:43.274 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.730156386999999, 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082147929999999997, 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 419 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:43.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502512+0000", 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.502514+0000", 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.65e-06 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503628+0000", 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001113239 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503646+0000", 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7906000000000001e-05 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503653+0000", 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2339999999999997e-06 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503741+0000", 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8436999999999994e-05 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.503820+0000", 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8751000000000005e-05 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505036+0000", 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001216017 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510097+0000", 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050614550000000003 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510625+0000", 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000527667 2024-09-16T04:08:43.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510727+0000", 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000101437 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:420 3.9s0 3:94de6fe1:::smithi01639990-3:head [append 0~491520] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.727524486, 2024-09-16T04:08:43.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089351980000000001, 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 420 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505144+0000", 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505614+0000", 2024-09-16T04:08:43.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00046960099999999999 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505627+0000", 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3275000000000001e-05 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505636+0000", 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2609999999999993e-06 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505739+0000", 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010322700000000001 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.505847+0000", 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107929 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.506958+0000", 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011114580000000001 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510232+0000", 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032740080000000001 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.513941+0000", 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0037089470000000002 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514079+0000", 2024-09-16T04:08:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000137744 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:423 3.9s0 3:94de6fe1:::smithi01639990-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.72505211, 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010412819, 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.286 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 423 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507617+0000", 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6599999999999996e-07 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507621+0000", 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9609999999999999e-06 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507628+0000", 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3139999999999998e-06 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507637+0000", 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1840000000000005e-06 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510428+0000", 2024-09-16T04:08:43.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027909060000000001 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510505+0000", 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6679000000000002e-05 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514954+0000", 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044494410000000002 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515004+0000", 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0244999999999998e-05 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518029+0000", 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030245229999999999 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:428 3.9s0 3:92dcccbe:::smithi01639990-16:head [rollback c] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:43.291 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.724989725, 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0081187840000000004, 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 428 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507678+0000", 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507679+0000", 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0900000000000002e-07 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507684+0000", 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3199999999999999e-06 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507691+0000", 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1160000000000001e-06 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507697+0000", 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3129999999999998e-06 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512167+0000", 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044708880000000001 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512254+0000", 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6329999999999995e-05 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.512662+0000", 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040792999999999998 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.295 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514513+0000", 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0018516019999999999 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515756+0000", 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001242976 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515797+0000", 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0899999999999998e-05 2024-09-16T04:08:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.724945624, 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:43.297 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.298 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:08:43.299 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.300 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:08:43.301 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.302 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.703232599, 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.304 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:08:43.305 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.306 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.307 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.703180372, 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:08:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:08:43.310 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.311 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:08:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:08:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:08:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:08:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.313 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.314 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.314 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.314 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 67/56 ECSubWrite(tid=320, reqid=osd.3.0:320, at_version=67'62, trim_to=0'0, roll_forward_to=66'60))", 2024-09-16T04:08:43.314 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:43.314 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.700119398, 2024-09-16T04:08:43.314 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01263539, 2024-09-16T04:08:43.314 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.314 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.314 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.315 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6899999999999999e-07 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532569+0000", 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0305999999999998e-05 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532682+0000", 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112361 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.316 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532692+0000", 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0832000000000001e-05 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532753+0000", 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0568e-05 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532763+0000", 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6010000000000001e-06 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.317 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545152+0000", 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012389908 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545184+0000", 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1644999999999999e-05 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 67/56 ECSubWrite(tid=321, reqid=osd.3.0:321, at_version=67'92, trim_to=0'0, roll_forward_to=66'90))", 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.532571+0000", 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.70009675199999999, 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012633584, 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532571+0000", 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.319 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532571+0000", 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532572+0000", 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.48e-07 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532583+0000", 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1116e-05 2024-09-16T04:08:43.320 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532796+0000", 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000213406 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532803+0000", 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5880000000000003e-06 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532860+0000", 2024-09-16T04:08:43.321 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7175000000000001e-05 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532868+0000", 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5329999999999994e-06 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545182+0000", 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012313945 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.322 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545205+0000", 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2572999999999999e-05 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.12s3 67/56 ECSubWrite(tid=130, reqid=osd.0.0:130, at_version=67'80, trim_to=0'0, roll_forward_to=65'78))", 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.532611+0000", 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.70005738399999995, 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012566290000000001, 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532611+0000", 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532611+0000", 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532611+0000", 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3300000000000001e-07 2024-09-16T04:08:43.324 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532626+0000", 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5356e-05 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532929+0000", 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030242099999999998 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532935+0000", 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2450000000000003e-06 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532981+0000", 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6150999999999999e-05 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532990+0000", 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7479999999999999e-06 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545137+0000", 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01214763 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545177+0000", 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9606e-05 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s1 67/56 ECSubWrite(tid=131, reqid=osd.0.0:131, at_version=67'31, trim_to=0'0, roll_forward_to=65'29))", 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.535232+0000", 2024-09-16T04:08:43.327 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.69743654899999996, 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010052236000000001, 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535232+0000", 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535232+0000", 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535232+0000", 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535252+0000", 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0026000000000001e-05 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535283+0000", 2024-09-16T04:08:43.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1015e-05 2024-09-16T04:08:43.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535292+0000", 2024-09-16T04:08:43.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7560000000000005e-06 2024-09-16T04:08:43.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535351+0000", 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8901999999999999e-05 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535358+0000", 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7490000000000001e-06 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545152+0000", 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0097946720000000008 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545284+0000", 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013136800000000001 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.16s4 67/56 ECSubWrite(tid=146, reqid=osd.5.0:146, at_version=67'24, trim_to=0'0, roll_forward_to=65'22))", 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.538736+0000", 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.69393255200000004, 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093344139999999992, 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.332 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.538736+0000", 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.538736+0000", 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.538736+0000", 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6400000000000001e-07 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.538759+0000", 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2898e-05 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.538790+0000", 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0414000000000001e-05 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.538797+0000", 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6899999999999992e-06 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.538857+0000", 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9605000000000003e-05 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.538867+0000", 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8400000000000007e-06 2024-09-16T04:08:43.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.548038+0000", 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0091708520000000002 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.548070+0000", 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2351000000000002e-05 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.337 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:43.338 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:43.338 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:43.336+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:43.439 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:43.437+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:43.441 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:43.441 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:43.441 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:43.441 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:43.450 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:43.532 INFO:tasks.rados.rados.0.smithi016.stdout:266: done (7 left) 2024-09-16T04:08:43.533 INFO:tasks.rados.rados.0.smithi016.stdout:267: read oid 48 snap 19 2024-09-16T04:08:43.533 INFO:tasks.rados.rados.0.smithi016.stdout:267: expect (ObjNum 100 snap 17 seq_num 100) 2024-09-16T04:08:43.533 INFO:tasks.rados.rados.0.smithi016.stdout:268: rmattr oid 44 current snap is 20 2024-09-16T04:08:43.533 INFO:tasks.rados.rados.0.smithi016.stdout:259: finishing write tid 3 to smithi01639990-18 2024-09-16T04:08:43.533 INFO:tasks.rados.rados.0.smithi016.stdout:259: finishing write tid 4 to smithi01639990-18 2024-09-16T04:08:43.533 INFO:tasks.rados.rados.0.smithi016.stdout:259: finishing write tid 5 to smithi01639990-18 2024-09-16T04:08:43.533 INFO:tasks.rados.rados.0.smithi016.stdout:259: finishing write tid 6 to smithi01639990-18 2024-09-16T04:08:43.533 INFO:tasks.rados.rados.0.smithi016.stdout:259: oid 18 updating version 0 to 59 2024-09-16T04:08:43.533 INFO:tasks.rados.rados.0.smithi016.stdout:259: oid 18 version 59 is already newer than 58 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:259: oid 18 version 59 is already newer than 57 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:259: oid 18 version 59 is already newer than 55 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:259: oid 18 version 59 is already newer than 56 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 18 v 59 (ObjNum 117 snap 20 seq_num 117) dirty exists 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:259: left oid 18 (ObjNum 117 snap 20 seq_num 117) 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:260: finishing write tid 1 to smithi01639990-9 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:260: finishing write tid 2 to smithi01639990-9 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:260: finishing write tid 3 to smithi01639990-9 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:260: finishing write tid 4 to smithi01639990-9 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:260: finishing write tid 5 to smithi01639990-9 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:260: finishing write tid 6 to smithi01639990-9 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:260: oid 9 updating version 0 to 56 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:260: oid 9 updating version 56 to 58 2024-09-16T04:08:43.534 INFO:tasks.rados.rados.0.smithi016.stdout:260: oid 9 version 58 is already newer than 57 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:260: oid 9 updating version 58 to 59 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:260: oid 9 updating version 59 to 60 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 9 v 60 (ObjNum 118 snap 20 seq_num 118) dirty exists 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:260: left oid 9 (ObjNum 118 snap 20 seq_num 118) 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:262: finishing write tid 1 to smithi01639990-22 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:264: finishing rollback tid 0 to smithi01639990-23 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:259: done (8 left) 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:260: done (7 left) 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 11 current snap is 20 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:263: finishing copy_from to smithi01639990-41 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:263: got expected ENOENT (src dne) 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:264: finishing rollback tid 1 to smithi01639990-23 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:263: finishing copy_from racing read to smithi01639990-41 2024-09-16T04:08:43.535 INFO:tasks.rados.rados.0.smithi016.stdout:264: finishing rollback tid 2 to smithi01639990-23 2024-09-16T04:08:43.536 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 23 v 27 (ObjNum 104 snap 17 seq_num 104) dirty exists 2024-09-16T04:08:43.536 INFO:tasks.rados.rados.0.smithi016.stdout:262: finishing write tid 2 to smithi01639990-22 2024-09-16T04:08:43.536 INFO:tasks.rados.rados.0.smithi016.stdout:261: expect (ObjNum 115 snap 19 seq_num 115) 2024-09-16T04:08:43.540 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:43.538+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:43.570 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:43.570 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:43.570 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:43.570 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:43.570 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:43.579 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:43.641 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:43.639+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:43.683 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:43.684 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:43.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:43.684 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:43.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.684 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:43.684 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 36.587131098999997, 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:43.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:43.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:43.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:43.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:43.689 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.689 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.689 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.690 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=155, reqid=osd.3.0:155, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:43.690 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:43.690 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.241798292999999, 2024-09-16T04:08:43.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073085160000000001, 2024-09-16T04:08:43.690 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.690 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.690 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.690 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:43.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1299999999999999e-07 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439663+0000", 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3030000000000001e-05 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439705+0000", 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1078999999999998e-05 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439714+0000", 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441461+0000", 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017466350000000001 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441475+0000", 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4404e-05 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446911+0000", 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054361430000000001 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446949+0000", 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7465000000000001e-05 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.694 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.237719525999999, 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068399539999999997, 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:43.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443719+0000", 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443720+0000", 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9500000000000001e-07 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444146+0000", 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042652899999999998 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444178+0000", 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1412999999999999e-05 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444185+0000", 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4270000000000002e-06 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444247+0000", 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1760000000000005e-05 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444254+0000", 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9260000000000003e-06 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450519+0000", 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062649960000000001 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450559+0000", 2024-09-16T04:08:43.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0108000000000003e-05 2024-09-16T04:08:43.699 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.699 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.699 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.699 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s5 42/27 ECSubWrite(tid=40, reqid=osd.0.0:40, at_version=42'11, trim_to=0'0, roll_forward_to=41'9))", 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.237545518000001, 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067397389999999998, 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:43.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443893+0000", 2024-09-16T04:08:43.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0700000000000001e-07 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443914+0000", 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1274e-05 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443937+0000", 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2991999999999999e-05 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.443945+0000", 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4780000000000002e-06 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444021+0000", 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5814000000000003e-05 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.444033+0000", 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2322e-05 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450586+0000", 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065529869999999997 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450633+0000", 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6764999999999997e-05 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.704 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.251614129, 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:43.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:08:43.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:08:43.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:08:43.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:08:43.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:08:43.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.791663902, 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:08:43.714 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.468151296999999, 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:08:43.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:08:43.718 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.210367438999999, 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078488229999999996, 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471072+0000", 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0279999999999999e-06 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471418+0000", 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034541800000000001 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471457+0000", 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471492+0000", 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:08:43.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471798+0000", 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030615599999999997 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471812+0000", 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.448e-05 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478881+0000", 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070685549999999998 2024-09-16T04:08:43.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478920+0000", 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9072000000000001e-05 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 53/27 ECSubWrite(tid=103, reqid=osd.5.0:103, at_version=53'29, trim_to=0'0, roll_forward_to=52'27))", 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.193810781, 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068898910000000004, 2024-09-16T04:08:43.723 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487628+0000", 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.22e-07 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487655+0000", 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.669e-05 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487696+0000", 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0930000000000003e-05 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487708+0000", 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2699e-05 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487808+0000", 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9557999999999998e-05 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.487819+0000", 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1297999999999999e-05 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.494471+0000", 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0066521230000000002 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.494518+0000", 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6371e-05 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.727 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 53/46 ECSubWrite(tid=213, reqid=osd.3.0:213, at_version=53'65, trim_to=0'0, roll_forward_to=52'63))", 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.191039768, 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070799060000000004, 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490399+0000", 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490424+0000", 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5196999999999999e-05 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490482+0000", 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7442000000000003e-05 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490499+0000", 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7467999999999999e-05 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490695+0000", 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000195897 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.490707+0000", 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2221999999999999e-05 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.497426+0000", 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067186169999999996 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.497479+0000", 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2876e-05 2024-09-16T04:08:43.731 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.732 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.732 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.732 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:43.732 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:43.732 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.241252971, 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084259260000000002, 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.91e-07 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440211+0000", 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4978999999999999e-05 2024-09-16T04:08:43.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440257+0000", 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6569000000000001e-05 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440268+0000", 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1056000000000001e-05 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440348+0000", 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9795000000000004e-05 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440364+0000", 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5996999999999999e-05 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448570+0000", 2024-09-16T04:08:43.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082059590000000005 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448611+0000", 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.138e-05 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.bs7 57/56 [PushOp(3:d683cb60:::smithi01639990-49:head, version: 53'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d683cb60:::smithi01639990-49:head@53'38, size: 2310144, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.120670799999999, 2024-09-16T04:08:43.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.043702234999999999, 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560769+0000", 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0049999999999999e-06 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.561301+0000", 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053276500000000004 2024-09-16T04:08:43.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603955+0000", 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042653605999999997 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603967+0000", 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2065000000000001e-05 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604063+0000", 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5599000000000004e-05 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604470+0000", 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040719500000000001 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=252, reqid=osd.3.0:252, at_version=57'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:43.741 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.771944659999999, 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27497091299999998, 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909517+0000", 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2886000000000001e-05 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183847+0000", 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27432981400000001 2024-09-16T04:08:43.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183853+0000", 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2929999999999997e-06 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183902+0000", 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9008999999999998e-05 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183911+0000", 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184435+0000", 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052493699999999995 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184465+0000", 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9292e-05 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=253, reqid=osd.3.0:253, at_version=57'67, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.771316628999999, 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50851210899999999, 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:43.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910123+0000", 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910136+0000", 2024-09-16T04:08:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.331e-05 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416910+0000", 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50677369000000005 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416919+0000", 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8290000000000003e-06 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416975+0000", 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5983999999999998e-05 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416985+0000", 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418599+0000", 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001614515 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418634+0000", 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4588999999999997e-05 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 64/56 ECSubWrite(tid=285, reqid=client.4440.0:535, at_version=0'0, trim_to=0'0, roll_forward_to=64'72))", 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.1398695600000002, 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012095003, 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.750 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541570+0000", 2024-09-16T04:08:43.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.173e-06 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541582+0000", 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2106999999999999e-05 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552877+0000", 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011294692 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552889+0000", 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1875000000000001e-05 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552958+0000", 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9510000000000004e-05 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552967+0000", 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6240000000000008e-06 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553631+0000", 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066445299999999996 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553664+0000", 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2568999999999999e-05 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 67/56 ECSubWrite(tid=321, reqid=osd.3.0:321, at_version=67'92, trim_to=0'0, roll_forward_to=66'90))", 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.534990+0000", 2024-09-16T04:08:43.754 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.1464486620000001, 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072246840000000003, 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.534990+0000", 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.534990+0000", 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.534990+0000", 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0800000000000001e-07 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535011+0000", 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1331999999999998e-05 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535098+0000", 2024-09-16T04:08:43.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6644999999999995e-05 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535104+0000", 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5079999999999996e-06 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535167+0000", 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3189999999999996e-05 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535176+0000", 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4020000000000008e-06 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542178+0000", 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070018789999999999 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542215+0000", 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6520000000000003e-05 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.758 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 67/56 ECSubWrite(tid=320, reqid=osd.3.0:320, at_version=67'62, trim_to=0'0, roll_forward_to=66'60))", 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.535013+0000", 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.146425115, 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0069814059999999999, 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535013+0000", 2024-09-16T04:08:43.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535013+0000", 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535014+0000", 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3099999999999999e-07 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535026+0000", 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2723e-05 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535131+0000", 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000104781 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535139+0000", 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.435e-06 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535184+0000", 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5389999999999997e-05 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535194+0000", 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0164e-05 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541952+0000", 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067577749999999997 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541995+0000", 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2907000000000001e-05 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 68/46 ECSubWrite(tid=65, reqid=osd.4.0:65, at_version=68'22, trim_to=0'0, roll_forward_to=67'20))", 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.140143302, 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010787494999999999, 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.763 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:43.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541313+0000", 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7431e-05 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541369+0000", 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6311999999999999e-05 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541376+0000", 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5729999999999996e-06 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544486+0000", 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031104150000000001 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544495+0000", 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0960000000000007e-06 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552010+0000", 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075148200000000002 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552083+0000", 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2661000000000001e-05 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 68/56 ECSubWrite(tid=329, reqid=osd.3.0:329, at_version=68'68, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.13868958200000001, 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093665009999999993, 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.768 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542750+0000", 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.23e-07 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542770+0000", 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0679999999999999e-05 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542842+0000", 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1391999999999997e-05 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542850+0000", 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2320000000000001e-06 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545061+0000", 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022114220000000002 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545070+0000", 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6780000000000006e-06 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552088+0000", 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070176939999999997 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552115+0000", 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7679999999999999e-05 2024-09-16T04:08:43.771 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es5 68/56 ECSubWrite(tid=110, reqid=osd.11.0:110, at_version=68'17, trim_to=0'0, roll_forward_to=63'15))", 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.545011+0000", 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.13642745100000001, 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0071061220000000003, 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545011+0000", 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545011+0000", 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545011+0000", 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.79e-07 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545032+0000", 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0414000000000001e-05 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545075+0000", 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3769000000000001e-05 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545084+0000", 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.456e-06 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545146+0000", 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2413e-05 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545168+0000", 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1331e-05 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552081+0000", 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0069133479999999997 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:43.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:43.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:43.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552117+0000", 2024-09-16T04:08:43.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6211999999999998e-05 2024-09-16T04:08:43.776 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.776 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.776 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.776 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:43.776 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:43.776 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:43.778 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:43.779 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:43.775+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:43.779 INFO:tasks.rados.rados.0.smithi016.stdout:262: finishing write tid 3 to smithi01639990-22 2024-09-16T04:08:43.779 INFO:tasks.rados.rados.0.smithi016.stdout:262: finishing write tid 4 to smithi01639990-22 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: finishing write tid 5 to smithi01639990-22 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:256: finishing rollback tid 2 to smithi01639990-16 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 16 v 64 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: finishing write tid 6 to smithi01639990-22 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: finishing write tid 7 to smithi01639990-22 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: oid 22 updating version 0 to 87 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: oid 22 updating version 87 to 89 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: oid 22 version 89 is already newer than 86 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: oid 22 version 89 is already newer than 88 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: oid 22 version 89 is already newer than 85 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: oid 22 updating version 89 to 90 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 22 v 90 (ObjNum 119 snap 20 seq_num 119) dirty exists 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:262: left oid 22 (ObjNum 119 snap 20 seq_num 119) 2024-09-16T04:08:43.780 INFO:tasks.rados.rados.0.smithi016.stdout:269: seq_num 120 ranges {1523712=475136,1998848=655360,2654208=622592,3276800=278528} 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:269: writing smithi01639990-11 from 1523712 to 1998848 tid 1 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:269: writing smithi01639990-11 from 1998848 to 2654208 tid 2 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:269: writing smithi01639990-11 from 2654208 to 3276800 tid 3 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:269: writing smithi01639990-11 from 3276800 to 3555328 tid 4 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:256: done (7 left) 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:261: done (6 left) 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:262: done (5 left) 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:263: done (4 left) 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:264: done (3 left) 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:270: setattr oid 1 current snap is 20 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 44 v 64 (ObjNum 109 snap 18 seq_num 109) dirty exists 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:267: expect (ObjNum 100 snap 17 seq_num 100) 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:269: finishing write tid 1 to smithi01639990-11 2024-09-16T04:08:43.781 INFO:tasks.rados.rados.0.smithi016.stdout:269: finishing write tid 2 to smithi01639990-11 2024-09-16T04:08:43.782 INFO:tasks.rados.rados.0.smithi016.stdout:269: finishing write tid 3 to smithi01639990-11 2024-09-16T04:08:43.782 INFO:tasks.rados.rados.0.smithi016.stdout:269: finishing write tid 4 to smithi01639990-11 2024-09-16T04:08:43.782 INFO:tasks.rados.rados.0.smithi016.stdout:267: done (3 left) 2024-09-16T04:08:43.782 INFO:tasks.rados.rados.0.smithi016.stdout:268: done (2 left) 2024-09-16T04:08:43.782 INFO:tasks.rados.rados.0.smithi016.stdout:271: rmattr oid 18 current snap is 20 2024-09-16T04:08:43.782 INFO:tasks.rados.rados.0.smithi016.stdout:271: done (2 left) 2024-09-16T04:08:43.782 INFO:tasks.rados.rados.0.smithi016.stdout:272: snap_remove snap 14 2024-09-16T04:08:43.879 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:43.877+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:43.923 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:43.923 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:43.923 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:43.923 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:43.932 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:43.980 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:43.978+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:44.033 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:44.034 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:44.034 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:44.034 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:44.034 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:44.042 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:44.081 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:44.079+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:44.146 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:44.146 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:44.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:44.146 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.040950690999999, 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083578109999999997, 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:44.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102823+0000", 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102824+0000", 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4899999999999995e-07 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102859+0000", 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4805999999999998e-05 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102891+0000", 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2394000000000002e-05 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.102898+0000", 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.204e-06 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103849+0000", 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00095089899999999995 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.103858+0000", 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3480000000000004e-06 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111149+0000", 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072911809999999999 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.111181+0000", 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2329999999999997e-05 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:44.151 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.450278+0000", 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.693496207999999, 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007464432, 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450278+0000", 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450278+0000", 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450279+0000", 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.031e-06 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450548+0000", 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000269659 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450626+0000", 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7577000000000006e-05 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450635+0000", 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2920000000000001e-06 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450868+0000", 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00023291100000000001 2024-09-16T04:08:44.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.450879+0000", 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0988999999999999e-05 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457700+0000", 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068211620000000004 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.457742+0000", 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1811000000000003e-05 2024-09-16T04:08:44.155 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 48/46 ECSubWrite(tid=190, reqid=osd.3.0:190, at_version=48'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.678129618, 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088361580000000002, 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465645+0000", 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4399999999999999e-07 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465670+0000", 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4875000000000002e-05 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465791+0000", 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012066900000000001 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465800+0000", 2024-09-16T04:08:44.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4539999999999996e-06 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465872+0000", 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2127999999999998e-05 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465881+0000", 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3129999999999999e-06 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474449+0000", 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085683760000000008 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474480+0000", 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1498999999999998e-05 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:44.160 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.475668+0000", 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.66810632, 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084809299999999994, 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475668+0000", 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475668+0000", 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475668+0000", 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3700000000000001e-07 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475747+0000", 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9559000000000002e-05 2024-09-16T04:08:44.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476148+0000", 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040095300000000001 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476157+0000", 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9870000000000005e-06 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476859+0000", 2024-09-16T04:08:44.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00070109199999999995 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476868+0000", 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1689999999999998e-06 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484115+0000", 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072471239999999998 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484149+0000", 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3708999999999999e-05 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 52/46 ECSubWrite(tid=210, reqid=client.4440.0:434, at_version=52'63, trim_to=0'0, roll_forward_to=50'60))", 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.626007092000002, 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83621405000000004, 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.165 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8200000000000001e-07 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517790+0000", 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2540000000000001e-05 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352789+0000", 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83499984500000002 2024-09-16T04:08:44.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352804+0000", 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4569999999999999e-05 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352928+0000", 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012396799999999999 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352943+0000", 2024-09-16T04:08:44.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353943+0000", 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099977500000000006 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353981+0000", 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7700999999999997e-05 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=217, reqid=client.4440.0:449, at_version=53'58, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.632497056, 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0096255799999999999, 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:44.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511278+0000", 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.068e-06 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511623+0000", 2024-09-16T04:08:44.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034493499999999999 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515041+0000", 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034178889999999999 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515048+0000", 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3479999999999998e-06 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515115+0000", 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6469000000000003e-05 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515126+0000", 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1187e-05 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520872+0000", 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057460280000000002 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520902+0000", 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0655999999999999e-05 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=218, reqid=client.4440.0:450, at_version=53'59, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.629152481, 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089217440000000005, 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.174 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514623+0000", 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2300000000000001e-06 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515021+0000", 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039790399999999998 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519384+0000", 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043635380000000001 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519391+0000", 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9709999999999998e-06 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519441+0000", 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9493999999999997e-05 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519449+0000", 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:08:44.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.523504+0000", 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040551220000000004 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.523543+0000", 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9271999999999999e-05 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=220, reqid=client.4440.0:452, at_version=53'61, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.624003539, 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51827822899999998, 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519771+0000", 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.61e-07 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519788+0000", 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6699999999999999e-05 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036882+0000", 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51709414300000001 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036895+0000", 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3363999999999999e-05 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036964+0000", 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8734999999999998e-05 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036973+0000", 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0289999999999994e-06 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038014+0000", 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010410529999999999 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038049+0000", 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4443999999999998e-05 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs3 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.446789+0000", 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.696984428, 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079296090000000007, 2024-09-16T04:08:44.183 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.446789+0000", 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.446789+0000", 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.446790+0000", 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.08e-07 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.446827+0000", 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7766999999999999e-05 2024-09-16T04:08:44.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.447541+0000", 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071359099999999996 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.447550+0000", 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8170000000000005e-06 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.449555+0000", 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0020049859999999998 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.449565+0000", 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0003999999999999e-05 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.454688+0000", 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0051229589999999998 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.454719+0000", 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1377000000000001e-05 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:525 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~540672] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.626881+0000", 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.5168932789999996, 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076914009999999996, 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.188 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 525 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.626881+0000", 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.626881+0000", 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.626882+0000", 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.798e-06 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628281+0000", 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0013982829999999999 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628293+0000", 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2170999999999999e-05 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628299+0000", 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.088e-06 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628341+0000", 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2449e-05 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.628454+0000", 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011251699999999999 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.629546+0000", 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001091828 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630981+0000", 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014352449999999999 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634515+0000", 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003533694 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634572+0000", 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7327999999999997e-05 2024-09-16T04:08:44.193 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:526 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~622592] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.5132071509999996, 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01502261, 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.194 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 526 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:44.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630568+0000", 2024-09-16T04:08:44.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5680000000000001e-06 2024-09-16T04:08:44.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631353+0000", 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078490700000000005 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631367+0000", 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3441000000000001e-05 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631375+0000", 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8809999999999996e-06 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631465+0000", 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0673000000000004e-05 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631565+0000", 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9753000000000005e-05 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.632641+0000", 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001076097 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634418+0000", 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017769680000000001 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645537+0000", 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011119393999999999 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645589+0000", 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1928000000000002e-05 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:527 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.512370078, 2024-09-16T04:08:44.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015053211, 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 527 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631405+0000", 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.08e-06 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631411+0000", 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9009999999999999e-06 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631419+0000", 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631426+0000", 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4999999999999996e-06 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633558+0000", 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002131796 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633663+0000", 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105833 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634106+0000", 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00044296899999999998 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.636380+0000", 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022740849999999999 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646383+0000", 2024-09-16T04:08:44.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01000238 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646457+0000", 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4258000000000006e-05 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:528 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.5122954970000002, 2024-09-16T04:08:44.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65958352099999995, 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 528 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631479+0000", 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9599999999999998e-07 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631483+0000", 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8160000000000004e-06 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631494+0000", 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0407999999999999e-05 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631501+0000", 2024-09-16T04:08:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2269999999999996e-06 2024-09-16T04:08:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634655+0000", 2024-09-16T04:08:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031544339999999998 2024-09-16T04:08:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634732+0000", 2024-09-16T04:08:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6495999999999994e-05 2024-09-16T04:08:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646864+0000", 2024-09-16T04:08:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012131807999999999 2024-09-16T04:08:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646965+0000", 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010173 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.291062+0000", 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64409670600000002 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 67/56 ECSubWrite(tid=321, reqid=osd.3.0:321, at_version=67'92, trim_to=0'0, roll_forward_to=66'90))", 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.533170+0000", 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.610603558, 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086084990000000004, 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533170+0000", 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533170+0000", 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533171+0000", 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.05e-07 2024-09-16T04:08:44.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533221+0000", 2024-09-16T04:08:44.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0498999999999999e-05 2024-09-16T04:08:44.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533284+0000", 2024-09-16T04:08:44.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2853000000000003e-05 2024-09-16T04:08:44.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533293+0000", 2024-09-16T04:08:44.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2529999999999993e-06 2024-09-16T04:08:44.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533356+0000", 2024-09-16T04:08:44.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3107000000000004e-05 2024-09-16T04:08:44.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533365+0000", 2024-09-16T04:08:44.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9390000000000003e-06 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541744+0000", 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083788960000000003 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541779+0000", 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4746999999999998e-05 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.19s3 67/56 ECSubWrite(tid=145, reqid=osd.5.0:142, at_version=0'0, trim_to=0'0, roll_forward_to=67'48))", 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.533259+0000", 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.6105153190000001, 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085526500000000002, 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533259+0000", 2024-09-16T04:08:44.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533259+0000", 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533259+0000", 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9500000000000001e-07 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533267+0000", 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8269999999999998e-06 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533413+0000", 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000145888 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533420+0000", 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3089999999999998e-06 2024-09-16T04:08:44.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533465+0000", 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5469999999999997e-05 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533475+0000", 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9920000000000002e-06 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541776+0000", 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083010389999999996 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541811+0000", 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4929999999999999e-05 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s7 67/46 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=67'20, trim_to=0'0, roll_forward_to=66'18))", 2024-09-16T04:08:44.224 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.6101601539999999, 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090747759999999997, 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8199999999999999e-07 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533635+0000", 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1378000000000001e-05 2024-09-16T04:08:44.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533659+0000", 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.374e-05 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533667+0000", 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2230000000000007e-06 2024-09-16T04:08:44.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533785+0000", 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117607 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533795+0000", 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0180000000000001e-05 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542641+0000", 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088456149999999994 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542689+0000", 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7851e-05 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s5 67/56 ECSubWrite(tid=131, reqid=osd.0.0:131, at_version=67'31, trim_to=0'0, roll_forward_to=65'29))", 2024-09-16T04:08:44.229 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.535353+0000", 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.608421356, 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073564119999999997, 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535353+0000", 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535353+0000", 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535353+0000", 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4199999999999996e-07 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535384+0000", 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0808e-05 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535399+0000", 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5366000000000002e-05 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535406+0000", 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6000000000000003e-06 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535455+0000", 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9471000000000001e-05 2024-09-16T04:08:44.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.535464+0000", 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9029999999999993e-06 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542675+0000", 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0072107209999999998 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542709+0000", 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4001000000000002e-05 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s2 68/56 ECSubWrite(tid=134, reqid=osd.0.0:134, at_version=68'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.60173917700000001, 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010826218, 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.234 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8599999999999998e-07 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542054+0000", 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8423999999999998e-05 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542081+0000", 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7806000000000001e-05 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542089+0000", 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5789999999999998e-06 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542147+0000", 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7978e-05 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542157+0000", 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:08:44.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552766+0000", 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010609143999999999 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552861+0000", 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4841999999999998e-05 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 68/56 ECSubWrite(tid=330, reqid=osd.3.0:330, at_version=68'75, trim_to=0'0, roll_forward_to=64'73))", 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.60068422099999996, 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094449849999999995, 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6000000000000003e-07 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543110+0000", 2024-09-16T04:08:44.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9695e-05 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543139+0000", 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8467999999999999e-05 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543148+0000", 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5349999999999993e-06 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543200+0000", 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1536999999999998e-05 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543206+0000", 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9379999999999997e-06 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552495+0000", 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.009289538 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552535+0000", 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9613999999999997e-05 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.es7 68/56 ECSubWrite(tid=110, reqid=osd.11.0:110, at_version=68'17, trim_to=0'0, roll_forward_to=63'15))", 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.544784+0000", 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.59898952500000002, 2024-09-16T04:08:44.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0081313670000000005, 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544784+0000", 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544784+0000", 2024-09-16T04:08:44.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544785+0000", 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3499999999999996e-07 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544804+0000", 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8614000000000002e-05 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544845+0000", 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1826999999999998e-05 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544854+0000", 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3399999999999998e-06 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544906+0000", 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2531999999999998e-05 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544915+0000", 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7490000000000004e-06 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552879+0000", 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0079639489999999997 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552916+0000", 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6721000000000003e-05 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.247 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.248 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.248 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.248 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.248 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:44.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:44.250 INFO:teuthology.orchestra.run.smithi016.stdout:fsck success 2024-09-16T04:08:44.251 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:44.251 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:44.247+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:44.251 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-16T04:08:44.252 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-16T04:08:44.252 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T04:08:44.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T04:08:44.294 INFO:tasks.ceph.osd.1:Started 2024-09-16T04:08:44.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:08:44.378 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:44.376+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:44.394 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:44.394 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:44.394 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:44.395 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:44.403 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:44.446 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:44.447 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:44.447 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T04:08:44.451 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:44.449+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:44.454 INFO:teuthology.orchestra.run.smithi016.stderr:no valid command found; 10 closest matches: 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:0 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:1 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:2 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:abort 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:assert 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:config diff 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:config diff get 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:config get 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:config help [] 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:config set ... 2024-09-16T04:08:44.455 INFO:teuthology.orchestra.run.smithi016.stderr:admin_socket: invalid command 2024-09-16T04:08:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:08:44.462 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-16T04:08:44.470 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:44.471 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:44.491 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:44.491 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:44.505 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:44.506 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:44.506 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:44.506 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:44.506 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:44.513 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:44.514 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:44.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:44.533 INFO:tasks.rados.rados.0.smithi016.stdout:272: done (2 left) 2024-09-16T04:08:44.534 INFO:tasks.rados.rados.0.smithi016.stdout:273: setattr oid 5 current snap is 20 2024-09-16T04:08:44.534 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 1 v 57 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-09-16T04:08:44.534 INFO:tasks.rados.rados.0.smithi016.stdout:269: finishing write tid 5 to smithi01639990-11 2024-09-16T04:08:44.534 INFO:tasks.rados.rados.0.smithi016.stdout:269: finishing write tid 6 to smithi01639990-11 2024-09-16T04:08:44.534 INFO:tasks.rados.rados.0.smithi016.stdout:269: oid 11 updating version 0 to 62 2024-09-16T04:08:44.534 INFO:tasks.rados.rados.0.smithi016.stdout:269: oid 11 version 62 is already newer than 61 2024-09-16T04:08:44.534 INFO:tasks.rados.rados.0.smithi016.stdout:269: oid 11 updating version 62 to 64 2024-09-16T04:08:44.534 INFO:tasks.rados.rados.0.smithi016.stdout:269: oid 11 version 64 is already newer than 63 2024-09-16T04:08:44.535 INFO:tasks.rados.rados.0.smithi016.stdout:269: oid 11 updating version 64 to 65 2024-09-16T04:08:44.535 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 11 v 65 (ObjNum 120 snap 20 seq_num 120) dirty exists 2024-09-16T04:08:44.535 INFO:tasks.rados.rados.0.smithi016.stdout:269: left oid 11 (ObjNum 120 snap 20 seq_num 120) 2024-09-16T04:08:44.535 INFO:tasks.rados.rados.0.smithi016.stdout:269: done (2 left) 2024-09-16T04:08:44.535 INFO:tasks.rados.rados.0.smithi016.stdout:270: done (1 left) 2024-09-16T04:08:44.535 INFO:tasks.rados.rados.0.smithi016.stdout:274: snap_create 2024-09-16T04:08:44.536 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:44.536 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:44.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:08:44.555 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 5 v 26 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:08:44.557 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:44.555+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:44.577 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:44.577 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:44.620 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:44.652 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:44.650+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:44.665 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:44.667 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:44.667 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:44.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:44.667 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:44.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.667 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:124 1.0 e22/19 1:30306672:devicehealth::main.db.0000000000000000:head v 22'86, mlcod=21'84)", 2024-09-16T04:08:44.667 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:44.667 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 52.233884891999999, 2024-09-16T04:08:44.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020095396000000001, 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430782+0000", 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9699999999999996e-07 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430789+0000", 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4660000000000002e-06 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450028+0000", 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019238816999999998 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450042+0000", 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4487e-05 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450169+0000", 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126787 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450188+0000", 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450816+0000", 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062741399999999997 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450877+0000", 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0782999999999997e-05 2024-09-16T04:08:44.671 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:197 3.es0 3:7f241f48:::smithi01639990-30:head [copy-from ver 23] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 36.454675543999997, 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.023242381999999999, 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.672 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 197 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6499999999999999e-07 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209995+0000", 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.388e-06 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210002+0000", 2024-09-16T04:08:44.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5209999999999997e-06 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210008+0000", 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8780000000000003e-06 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8421999999999999e-05 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210099+0000", 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2469999999999997e-05 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227450+0000", 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017351235 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232352+0000", 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049017239999999997 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233168+0000", 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081616600000000001 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233233+0000", 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5012999999999996e-05 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:198 3.es0 3:7f241f48:::smithi01639990-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:44.677 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 36.454645235000001, 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019673758, 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 198 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3099999999999998e-07 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210024+0000", 2024-09-16T04:08:44.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2620000000000001e-06 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210030+0000", 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9020000000000002e-06 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210037+0000", 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1480000000000003e-06 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210227+0000", 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00019055499999999999 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210269+0000", 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2246999999999997e-05 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229480+0000", 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019210896000000002 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229587+0000", 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107026 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229695+0000", 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107291 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:213 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:44.682 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.548566729999997, 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53275775299999995, 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 213 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116100+0000", 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116103+0000", 2024-09-16T04:08:44.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.174e-06 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116109+0000", 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4330000000000003e-06 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116114+0000", 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1399999999999999e-06 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119298+0000", 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031846729999999998 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119365+0000", 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6484999999999998e-05 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119675+0000", 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00031021999999999999 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122365+0000", 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002689851 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648785+0000", 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52641971300000001 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648857+0000", 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2408000000000002e-05 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.687 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:214 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.548544053999997, 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53416417900000002, 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 214 2024-09-16T04:08:44.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2100000000000002e-07 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116125+0000", 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2649999999999999e-06 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116132+0000", 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5190000000000003e-06 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116137+0000", 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9869999999999999e-06 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120036+0000", 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038984979999999998 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120118+0000", 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2312999999999996e-05 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649191+0000", 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52907320199999996 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649279+0000", 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7585999999999998e-05 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.650286+0000", 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010073879999999999 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.692 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "replica_scrub(pg: 3.4s1,from:0'0,to:37'17,epoch:47/46,start:3:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.806742517, 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16799329900000001, 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:44.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9400000000000005e-07 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857929+0000", 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4719999999999997e-06 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025793+0000", 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16786378299999999 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025803+0000", 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9629999999999993e-06 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025875+0000", 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2914e-05 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025887+0000", 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1345999999999999e-05 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025917+0000", 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0127000000000001e-05 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.696 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.4s1,RELEASE,e:47,from:1(0),reservation_nonce:0)", 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.637597542000002, 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26135687200000002, 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2800000000000005e-07 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027075+0000", 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3859999999999998e-06 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288302+0000", 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26122710999999998 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288308+0000", 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6189999999999999e-06 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288392+0000", 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4549000000000007e-05 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288414+0000", 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1576000000000001e-05 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288425+0000", 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1704e-05 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.700 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:385 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.198595269999998, 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01911771, 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.701 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 385 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466073+0000", 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4389999999999999e-06 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466084+0000", 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1157e-05 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466101+0000", 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6334999999999999e-05 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466111+0000", 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466255+0000", 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014411 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466263+0000", 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.858e-06 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474564+0000", 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083014019999999994 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474734+0000", 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016996599999999999 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475786+0000", 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010515909999999999 2024-09-16T04:08:44.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.482124+0000", 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063382530000000003 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485117+0000", 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029928120000000001 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485189+0000", 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1868999999999999e-05 2024-09-16T04:08:44.706 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:386 3.8s0 3:17b72cd6:::smithi01639990-47:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.198523537, 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019422101000000001, 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.707 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 386 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8100000000000002e-07 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466147+0000", 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6559999999999998e-06 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466155+0000", 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6240000000000002e-06 2024-09-16T04:08:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466162+0000", 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4850000000000003e-06 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466343+0000", 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000181077 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466349+0000", 2024-09-16T04:08:44.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1229999999999997e-06 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476407+0000", 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010057885000000001 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476516+0000", 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108738 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477098+0000", 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000582024 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484050+0000", 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0069524950000000004 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485519+0000", 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014691280000000001 2024-09-16T04:08:44.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485565+0000", 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5185000000000002e-05 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:387 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.198495436000002, 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51761721800000005, 2024-09-16T04:08:44.713 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 387 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6399999999999998e-07 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466173+0000", 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.396e-06 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466179+0000", 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9290000000000003e-06 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466184+0000", 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9760000000000003e-06 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466397+0000", 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021269400000000001 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466406+0000", 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0769999999999997e-06 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479302+0000", 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012896014000000001 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479415+0000", 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112767 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479914+0000", 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00049861199999999995 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484063+0000", 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004149158 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983716+0000", 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49965376500000003 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983788+0000", 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1465999999999996e-05 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/46 ECSubWriteReply(tid=59, last_complete=48'38, committed=1, applied=1))", 2024-09-16T04:08:44.719 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.180454424000001, 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50006588100000005, 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2499999999999999e-07 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484214+0000", 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6440000000000001e-06 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983573+0000", 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49935939200000001 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983581+0000", 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983628+0000", 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7642999999999998e-05 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.984278+0000", 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00064939900000000005 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "rep_scrubmap(3.8s0 e46 from shard 10(6))", 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.678353132000002, 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21666559099999999, 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.723 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986314+0000", 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5600000000000005e-07 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986319+0000", 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0950000000000003e-06 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202862+0000", 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21654300900000001 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202872+0000", 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0077e-05 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202931+0000", 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9568000000000003e-05 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202942+0000", 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0139999999999999e-05 2024-09-16T04:08:44.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202979+0000", 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6946000000000002e-05 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s3 53/48 ECSubWrite(tid=39, reqid=osd.4.0:39, at_version=53'24, trim_to=0'0, roll_forward_to=51'22))", 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.175728343999999, 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017845360000000001, 2024-09-16T04:08:44.727 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2000000000000001e-07 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488964+0000", 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6418e-05 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502624+0000", 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013659439000000001 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502636+0000", 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2401999999999999e-05 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502702+0000", 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6006999999999993e-05 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502712+0000", 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8870000000000004e-06 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506740+0000", 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040279319999999997 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506783+0000", 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3055e-05 2024-09-16T04:08:44.731 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:443 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~425984] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.495031+0000", 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.169634813999998, 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011905696, 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.732 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 443 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.495031+0000", 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.495031+0000", 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.495038+0000", 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.443e-06 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498278+0000", 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0032401050000000001 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498294+0000", 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6531000000000001e-05 2024-09-16T04:08:44.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498304+0000", 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3130000000000004e-06 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498378+0000", 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4790999999999996e-05 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.498451+0000", 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2082000000000005e-05 2024-09-16T04:08:44.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.499529+0000", 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001078648 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506272+0000", 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067422960000000001 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506846+0000", 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00057419700000000003 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506937+0000", 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1290000000000002e-05 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:446 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.161126684999999, 2024-09-16T04:08:44.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67862900900000001, 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 446 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503541+0000", 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3680000000000001e-06 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503545+0000", 2024-09-16T04:08:44.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.561e-06 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503553+0000", 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8939999999999993e-06 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503559+0000", 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.224e-06 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505239+0000", 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001679739 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505303+0000", 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4219000000000003e-05 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505673+0000", 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000369374 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.507082+0000", 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014090330000000001 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182055+0000", 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67497357700000005 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182168+0000", 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011302 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.742 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:447 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.161097178999999, 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.680085987, 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 447 2024-09-16T04:08:44.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503570+0000", 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.44e-07 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503576+0000", 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4559999999999997e-06 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503602+0000", 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6264000000000001e-05 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503610+0000", 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4109999999999999e-06 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505945+0000", 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023353390000000001 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506004+0000", 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9410999999999998e-05 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182603+0000", 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67659811199999997 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182695+0000", 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1951000000000001e-05 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.183655+0000", 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00096039900000000002 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.747 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.156024854000002, 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.299096274, 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:44.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508642+0000", 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5400000000000002e-07 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508645+0000", 2024-09-16T04:08:44.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9249999999999999e-06 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807520+0000", 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.29887539899999999 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807534+0000", 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3643e-05 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807681+0000", 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014696699999999999 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807738+0000", 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6885999999999997e-05 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:44.751 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.155643389000002, 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67354323000000005, 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:44.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9500000000000002e-07 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509027+0000", 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4429999999999999e-06 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181881+0000", 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67285377899999999 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181891+0000", 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0479000000000001e-05 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181961+0000", 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0370000000000006e-05 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182566+0000", 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060456400000000003 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:520 3.es0 3:7f241f48:::smithi01639990-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.620430+0000", 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 7.0442363720000003, 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013388690999999999, 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:44.755 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 520 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620430+0000", 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620430+0000", 2024-09-16T04:08:44.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620431+0000", 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4399999999999998e-07 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620435+0000", 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1529999999999999e-06 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620445+0000", 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0458999999999999e-05 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.620451+0000", 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0519999999999998e-06 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.623070+0000", 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0026187659999999998 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.623129+0000", 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8658e-05 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.629860+0000", 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067308730000000001 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.629972+0000", 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011219599999999999 2024-09-16T04:08:44.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633818+0000", 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00384659 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.4s1 62/56 ECSubRead(tid=8, to_read={3:26eb5942:::smithi01639990-23:head=0,4096,0}, subchunks={3:26eb5942:::smithi01639990-23:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 7.0170125590000003, 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64312231099999995, 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:44.760 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2500000000000004e-07 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647678+0000", 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3421000000000002e-05 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290437+0000", 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64275884900000002 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290450+0000", 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2858e-05 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290545+0000", 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5463000000000004e-05 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290776+0000", 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000230895 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.763 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.764 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:44.764 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:44.764 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:44.765 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:08:44.766 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:44.766 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:44.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:08:44.767 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:44.763+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:44.866 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:44.864+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:44.913 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:44.913 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:44.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:44.967 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:44.965+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:45.029 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:45.029 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:45.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:45.081 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:45.079+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:45.144 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:45.144 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:45.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:45.168 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:45.167+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:45.270 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:45.268+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:45.300 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:45.301 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:45.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:45.370 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:45.369+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:45.417 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:45.417 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:45.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:45.432 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-16T04:08:45.435 INFO:tasks.rados.rados.0.smithi016.stdout:273: done (1 left) 2024-09-16T04:08:45.435 INFO:tasks.rados.rados.0.smithi016.stdout:274: done (0 left) 2024-09-16T04:08:45.436 INFO:tasks.rados.rados.0.smithi016.stdout:275: delete oid 16 current snap is 21 2024-09-16T04:08:45.445 INFO:tasks.rados.rados.0.smithi016.stdout:275: done (0 left) 2024-09-16T04:08:45.445 INFO:tasks.rados.rados.0.smithi016.stdout:276: copy_from oid 28 from oid 44 current snap is 21 2024-09-16T04:08:45.445 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 48 current snap is 21 2024-09-16T04:08:45.445 INFO:tasks.rados.rados.0.smithi016.stdout:277: seq_num 122 ranges {1818624=507904,2326528=425984,2752512=442368,3194880=540672,3735552=147456} 2024-09-16T04:08:45.458 INFO:tasks.rados.rados.0.smithi016.stdout:277: writing smithi01639990-48 from 1818624 to 2326528 tid 1 2024-09-16T04:08:45.461 INFO:tasks.rados.rados.0.smithi016.stdout:277: writing smithi01639990-48 from 2326528 to 2752512 tid 2 2024-09-16T04:08:45.465 INFO:tasks.rados.rados.0.smithi016.stdout:277: writing smithi01639990-48 from 2752512 to 3194880 tid 3 2024-09-16T04:08:45.468 INFO:tasks.rados.rados.0.smithi016.stdout:277: writing smithi01639990-48 from 3194880 to 3735552 tid 4 2024-09-16T04:08:45.469 INFO:tasks.rados.rados.0.smithi016.stdout:277: writing smithi01639990-48 from 3735552 to 3883008 tid 5 2024-09-16T04:08:45.469 INFO:tasks.rados.rados.0.smithi016.stdout:278: delete oid 37 current snap is 21 2024-09-16T04:08:45.469 INFO:tasks.rados.rados.0.smithi016.stdout:276: finishing copy_from racing read to smithi01639990-28 2024-09-16T04:08:45.471 INFO:tasks.rados.rados.0.smithi016.stdout:276: finishing copy_from to smithi01639990-28 2024-09-16T04:08:45.471 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 28 v 64 (ObjNum 109 snap 18 seq_num 109) dirty exists 2024-09-16T04:08:45.471 INFO:tasks.rados.rados.0.smithi016.stdout:277: finishing write tid 1 to smithi01639990-48 2024-09-16T04:08:45.472 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:45.469+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:45.472 INFO:tasks.rados.rados.0.smithi016.stdout:277: finishing write tid 2 to smithi01639990-48 2024-09-16T04:08:45.472 INFO:tasks.rados.rados.0.smithi016.stdout:277: finishing write tid 3 to smithi01639990-48 2024-09-16T04:08:45.476 INFO:tasks.rados.rados.0.smithi016.stdout:277: finishing write tid 4 to smithi01639990-48 2024-09-16T04:08:45.477 INFO:tasks.rados.rados.0.smithi016.stdout:276: done (2 left) 2024-09-16T04:08:45.477 INFO:tasks.rados.rados.0.smithi016.stdout:278: done (1 left) 2024-09-16T04:08:45.477 INFO:tasks.rados.rados.0.smithi016.stdout:279: setattr oid 18 current snap is 21 2024-09-16T04:08:45.477 INFO:tasks.rados.rados.0.smithi016.stdout:277: finishing write tid 5 to smithi01639990-48 2024-09-16T04:08:45.477 INFO:tasks.rados.rados.0.smithi016.stdout:277: finishing write tid 6 to smithi01639990-48 2024-09-16T04:08:45.478 INFO:tasks.rados.rados.0.smithi016.stdout:277: finishing write tid 7 to smithi01639990-48 2024-09-16T04:08:45.479 INFO:tasks.rados.rados.0.smithi016.stdout:277: oid 48 updating version 0 to 83 2024-09-16T04:08:45.479 INFO:tasks.rados.rados.0.smithi016.stdout:277: oid 48 version 83 is already newer than 82 2024-09-16T04:08:45.479 INFO:tasks.rados.rados.0.smithi016.stdout:277: oid 48 updating version 83 to 85 2024-09-16T04:08:45.479 INFO:tasks.rados.rados.0.smithi016.stdout:277: oid 48 version 85 is already newer than 80 2024-09-16T04:08:45.479 INFO:tasks.rados.rados.0.smithi016.stdout:277: oid 48 version 85 is already newer than 84 2024-09-16T04:08:45.479 INFO:tasks.rados.rados.0.smithi016.stdout:277: oid 48 version 85 is already newer than 81 2024-09-16T04:08:45.479 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 48 v 85 (ObjNum 122 snap 21 seq_num 122) dirty exists 2024-09-16T04:08:45.479 INFO:tasks.rados.rados.0.smithi016.stdout:277: left oid 48 (ObjNum 122 snap 21 seq_num 122) 2024-09-16T04:08:45.480 INFO:tasks.rados.rados.0.smithi016.stdout:277: done (1 left) 2024-09-16T04:08:45.480 INFO:tasks.rados.rados.0.smithi016.stdout:280: copy_from oid 7 from oid 11 current snap is 21 2024-09-16T04:08:45.480 INFO:tasks.rados.rados.0.smithi016.stdout:281: read oid 49 snap -1 2024-09-16T04:08:45.480 INFO:tasks.rados.rados.0.smithi016.stdout:281: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-16T04:08:45.480 INFO:tasks.rados.rados.0.smithi016.stdout:282: setattr oid 27 current snap is 21 2024-09-16T04:08:45.485 INFO:tasks.rados.rados.0.smithi016.stdout:283: snap_create 2024-09-16T04:08:45.486 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 27 v 22 (ObjNum 33 snap 0 seq_num 2170924144) dirty exists 2024-09-16T04:08:45.488 INFO:tasks.rados.rados.0.smithi016.stdout:281: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-16T04:08:45.554 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:45.554 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:45.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:45.567 INFO:tasks.rados.rados.0.smithi016.stdout:280: finishing copy_from racing read to smithi01639990-7 2024-09-16T04:08:45.567 INFO:tasks.rados.rados.0.smithi016.stdout:280: finishing copy_from to smithi01639990-7 2024-09-16T04:08:45.567 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 7 v 65 (ObjNum 120 snap 20 seq_num 120) dirty exists 2024-09-16T04:08:45.573 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:45.571+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:45.673 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:45.671+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:45.674 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:45.674 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:45.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:45.696 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 18 v 69 (ObjNum 117 snap 20 seq_num 117) dirty exists 2024-09-16T04:08:45.759 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:45.757+0000 7fea06ae8640 -1 Falling back to public interface 2024-09-16T04:08:45.774 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:45.772+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:45.790 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:45.790 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:45.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:45.875 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:45.874+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:45.945 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:45.945 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:45.954 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:45.977 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:45.976+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:46.064 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:46.064 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:46.073 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:46.116 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:46.114+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:46.176 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:46.176 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:46.180 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:46.178+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:46.184 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:46.281 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:46.279+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:46.327 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:46.327 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:46.336 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:46.381 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:46.379+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:46.436 INFO:tasks.rados.rados.0.smithi016.stdout:279: done (4 left) 2024-09-16T04:08:46.437 INFO:tasks.rados.rados.0.smithi016.stdout:280: done (3 left) 2024-09-16T04:08:46.437 INFO:tasks.rados.rados.0.smithi016.stdout:281: done (2 left) 2024-09-16T04:08:46.437 INFO:tasks.rados.rados.0.smithi016.stdout:282: done (1 left) 2024-09-16T04:08:46.437 INFO:tasks.rados.rados.0.smithi016.stdout:283: done (0 left) 2024-09-16T04:08:46.437 INFO:tasks.rados.rados.0.smithi016.stdout:284: delete oid 17 current snap is 22 2024-09-16T04:08:46.441 INFO:tasks.rados.rados.0.smithi016.stdout:284: done (0 left) 2024-09-16T04:08:46.441 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 21 current snap is 22 2024-09-16T04:08:46.441 INFO:tasks.rados.rados.0.smithi016.stdout:285: seq_num 125 ranges {344064=557056,901120=475136,1376256=507904,1884160=475136,2359296=98304} 2024-09-16T04:08:46.445 INFO:tasks.rados.rados.0.smithi016.stdout:285: writing smithi01639990-21 from 344064 to 901120 tid 1 2024-09-16T04:08:46.448 INFO:tasks.rados.rados.0.smithi016.stdout:285: writing smithi01639990-21 from 901120 to 1376256 tid 2 2024-09-16T04:08:46.451 INFO:tasks.rados.rados.0.smithi016.stdout:285: writing smithi01639990-21 from 1376256 to 1884160 tid 3 2024-09-16T04:08:46.454 INFO:tasks.rados.rados.0.smithi016.stdout:285: writing smithi01639990-21 from 1884160 to 2359296 tid 4 2024-09-16T04:08:46.455 INFO:tasks.rados.rados.0.smithi016.stdout:285: writing smithi01639990-21 from 2359296 to 2457600 tid 5 2024-09-16T04:08:46.455 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 3 current snap is 22 2024-09-16T04:08:46.456 INFO:tasks.rados.rados.0.smithi016.stdout:285: finishing write tid 1 to smithi01639990-21 2024-09-16T04:08:46.456 INFO:tasks.rados.rados.0.smithi016.stdout:286: seq_num 126 ranges {3063808=573440,3637248=622592,4259840=622592,4882432=311296} 2024-09-16T04:08:46.475 INFO:tasks.rados.rados.0.smithi016.stdout:286: writing smithi01639990-3 from 3063808 to 3637248 tid 1 2024-09-16T04:08:46.479 INFO:tasks.rados.rados.0.smithi016.stdout:286: writing smithi01639990-3 from 3637248 to 4259840 tid 2 2024-09-16T04:08:46.482 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:46.480+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:46.482 INFO:tasks.rados.rados.0.smithi016.stdout:286: writing smithi01639990-3 from 4259840 to 4882432 tid 3 2024-09-16T04:08:46.485 INFO:tasks.rados.rados.0.smithi016.stdout:286: writing smithi01639990-3 from 4882432 to 5193728 tid 4 2024-09-16T04:08:46.485 INFO:tasks.rados.rados.0.smithi016.stdout:287: setattr oid 13 current snap is 22 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: finishing write tid 2 to smithi01639990-21 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: finishing write tid 3 to smithi01639990-21 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: finishing write tid 4 to smithi01639990-21 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: finishing write tid 5 to smithi01639990-21 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: finishing write tid 6 to smithi01639990-21 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: finishing write tid 7 to smithi01639990-21 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: oid 21 updating version 0 to 78 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: oid 21 version 78 is already newer than 74 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: oid 21 version 78 is already newer than 75 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: oid 21 version 78 is already newer than 76 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: oid 21 version 78 is already newer than 77 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:285: oid 21 updating version 78 to 79 2024-09-16T04:08:46.486 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 21 v 79 (ObjNum 125 snap 22 seq_num 125) dirty exists 2024-09-16T04:08:46.487 INFO:tasks.rados.rados.0.smithi016.stdout:285: left oid 21 (ObjNum 125 snap 22 seq_num 125) 2024-09-16T04:08:46.487 INFO:tasks.rados.rados.0.smithi016.stdout:286: finishing write tid 1 to smithi01639990-3 2024-09-16T04:08:46.487 INFO:tasks.rados.rados.0.smithi016.stdout:285: done (2 left) 2024-09-16T04:08:46.487 INFO:tasks.rados.rados.0.smithi016.stdout:288: snap_create 2024-09-16T04:08:46.487 INFO:tasks.rados.rados.0.smithi016.stdout:286: finishing write tid 2 to smithi01639990-3 2024-09-16T04:08:46.489 INFO:tasks.rados.rados.0.smithi016.stdout:286: finishing write tid 3 to smithi01639990-3 2024-09-16T04:08:46.489 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:46.489 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:46.490 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 13 v 43 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T04:08:46.493 INFO:tasks.rados.rados.0.smithi016.stdout:286: finishing write tid 4 to smithi01639990-3 2024-09-16T04:08:46.493 INFO:tasks.rados.rados.0.smithi016.stdout:286: finishing write tid 5 to smithi01639990-3 2024-09-16T04:08:46.493 INFO:tasks.rados.rados.0.smithi016.stdout:286: finishing write tid 6 to smithi01639990-3 2024-09-16T04:08:46.493 INFO:tasks.rados.rados.0.smithi016.stdout:286: oid 3 updating version 0 to 76 2024-09-16T04:08:46.494 INFO:tasks.rados.rados.0.smithi016.stdout:286: oid 3 updating version 76 to 78 2024-09-16T04:08:46.494 INFO:tasks.rados.rados.0.smithi016.stdout:286: oid 3 version 78 is already newer than 75 2024-09-16T04:08:46.494 INFO:tasks.rados.rados.0.smithi016.stdout:286: oid 3 version 78 is already newer than 74 2024-09-16T04:08:46.494 INFO:tasks.rados.rados.0.smithi016.stdout:286: oid 3 version 78 is already newer than 77 2024-09-16T04:08:46.494 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 3 v 78 (ObjNum 126 snap 22 seq_num 126) dirty exists 2024-09-16T04:08:46.494 INFO:tasks.rados.rados.0.smithi016.stdout:286: left oid 3 (ObjNum 126 snap 22 seq_num 126) 2024-09-16T04:08:46.499 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:46.584 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:46.581+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:46.642 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:46.642 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:46.651 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:46.684 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:46.682+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:46.755 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:46.755 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:46.764 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:46.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:46.813 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:46.811+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:46.867 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:46.868 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:46.876 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:46.885 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:46.884+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T04:08:46.929 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:46.929 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:46.949 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:46.949 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:46.971 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:46.971 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:46.981 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:46.981 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:46.985 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:46.984+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:46.988 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:47.086 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:47.085+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:47.088 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:47.088 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:47.091 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:47.092 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:47.101 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:47.109 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:47.109 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:47.130 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:47.130 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:47.153 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:47.153 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:47.176 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:47.187 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:47.185+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:47.197 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:47.198 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:47.220 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:08:47.223 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:08:47.245 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:47.245 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:47.245 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:47.245 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:47.254 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:47.298 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:47.297+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:47.357 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:47.358 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:47.358 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:47.358 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:47.358 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:47.366 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:47.388 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:47.387+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:47.439 INFO:tasks.rados.rados.0.smithi016.stdout:286: done (2 left) 2024-09-16T04:08:47.439 INFO:tasks.rados.rados.0.smithi016.stdout:287: done (1 left) 2024-09-16T04:08:47.439 INFO:tasks.rados.rados.0.smithi016.stdout:288: done (0 left) 2024-09-16T04:08:47.439 INFO:tasks.rados.rados.0.smithi016.stdout:289: snap_remove snap 21 2024-09-16T04:08:47.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:08:47.490 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:47.488+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:47.514 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.571139156999998, 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:47.515 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:47.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 56.588734914, 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:47.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:47.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.527 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 56.578372408, 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:47.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:47.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 55.081784356, 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:47.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:47.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.627209862999997, 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:47.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:47.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.543 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.627155801999997, 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:47.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:47.547 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.548 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.549 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.402887264, 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.550 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.551 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:47.552 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.824953303999997, 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:47.553 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.554 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.555 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:47.556 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.557 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s2 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.412179074999997, 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010866819, 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.558 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099684+0000", 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.004e-06 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099714+0000", 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9807000000000001e-05 2024-09-16T04:08:47.559 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099769+0000", 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5306000000000002e-05 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099781+0000", 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2091e-05 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099835+0000", 2024-09-16T04:08:47.560 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4058000000000003e-05 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099845+0000", 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5729999999999997e-06 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110509+0000", 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010664339 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110550+0000", 2024-09-16T04:08:47.561 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640999999999999e-05 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.408140158999998, 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.562 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.563 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:47.565 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.566 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.406752744999999, 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.567 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.568 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:47.569 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:47.570 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=188, reqid=client.4440.0:379, at_version=47'56, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.059621287999999, 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011758437, 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.571 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.572 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452266+0000", 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4151e-05 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459768+0000", 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075025179999999997 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459778+0000", 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0244999999999999e-05 2024-09-16T04:08:47.573 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459835+0000", 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6394999999999998e-05 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459845+0000", 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0518e-05 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463963+0000", 2024-09-16T04:08:47.574 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041182450000000004 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463999+0000", 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5667999999999998e-05 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 25.047433715, 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:08:47.575 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.576 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:08:47.577 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:08:47.578 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.579 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.074563765000001, 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019153804, 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.580 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437299+0000", 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437320+0000", 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1121999999999999e-05 2024-09-16T04:08:47.581 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437388+0000", 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7889e-05 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437397+0000", 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4390000000000006e-06 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441658+0000", 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042607110000000004 2024-09-16T04:08:47.582 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441667+0000", 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456416+0000", 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014748493 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456452+0000", 2024-09-16T04:08:47.583 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6159999999999999e-05 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.071374054, 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017174697999999999, 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.584 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440489+0000", 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.015e-06 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.585 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440509+0000", 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0338000000000001e-05 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440547+0000", 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440557+0000", 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7820000000000005e-06 2024-09-16T04:08:47.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442068+0000", 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015112179999999999 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442078+0000", 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4620000000000002e-06 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457639+0000", 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015560954 2024-09-16T04:08:47.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457663+0000", 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4121000000000002e-05 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=82, reqid=osd.8.0:81, at_version=0'0, trim_to=0'0, roll_forward_to=57'51))", 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.051649734, 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010694719, 2024-09-16T04:08:47.588 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460212+0000", 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.589 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460213+0000", 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4999999999999999e-07 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460222+0000", 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2669999999999995e-06 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.460651+0000", 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00042901799999999998 2024-09-16T04:08:47.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.462020+0000", 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013696680000000001 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.466451+0000", 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0044306529999999997 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.466460+0000", 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:08:47.591 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.470859+0000", 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043990979999999997 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.470907+0000", 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7679000000000002e-05 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.592 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.as7 68/56 ECSubWrite(tid=57, reqid=osd.6.0:57, at_version=68'42, trim_to=0'0, roll_forward_to=64'40))", 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9777068020000002, 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017528866000000001, 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.49e-07 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534179+0000", 2024-09-16T04:08:47.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3308000000000001e-05 2024-09-16T04:08:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546651+0000", 2024-09-16T04:08:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012471873 2024-09-16T04:08:47.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546660+0000", 2024-09-16T04:08:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2229999999999996e-06 2024-09-16T04:08:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546730+0000", 2024-09-16T04:08:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0123000000000006e-05 2024-09-16T04:08:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546741+0000", 2024-09-16T04:08:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0684e-05 2024-09-16T04:08:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551646+0000", 2024-09-16T04:08:47.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049059180000000004 2024-09-16T04:08:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551684+0000", 2024-09-16T04:08:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7588e-05 2024-09-16T04:08:47.599 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 68/56 ECSubWriteReply(tid=200, last_complete=68'24, committed=1, applied=1))", 2024-09-16T04:08:47.600 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:08:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.9486947780000001, 2024-09-16T04:08:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46574167999999999, 2024-09-16T04:08:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.601 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:08:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:08:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563168+0000", 2024-09-16T04:08:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4099999999999997e-07 2024-09-16T04:08:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563174+0000", 2024-09-16T04:08:47.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.327e-06 2024-09-16T04:08:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028683+0000", 2024-09-16T04:08:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46550948399999997 2024-09-16T04:08:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028698+0000", 2024-09-16T04:08:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4576000000000001e-05 2024-09-16T04:08:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028858+0000", 2024-09-16T04:08:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016062100000000001 2024-09-16T04:08:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028909+0000", 2024-09-16T04:08:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0730999999999999e-05 2024-09-16T04:08:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.608 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:654 3.ds0 3:bc502819:::smithi01639990-5:head [create,setxattr XXWdXV_kiPaajZX\\PgmqNaZqSVmab^hrtRYeagZr (1894) in=40b,setxattr NdZr`e\\^kXiksh]\\XocenQrhYc]c`[proOfo`Pd^ (33) in=40b,setxattr _`NqSdldlQbUNl^ZNRba\\naZhhdraiOgpafrZb[o (103) in=40b,setxattr iadimrq_[jOPUblhnVqTYqQNedg (1647) in=27b,setxattr `e\\rtQi]nmoP\\cm]oRYN`[]^qVUTsU^_oXm]h\\ec (6452) in=40b,setxattr ppZXc_dYi_WWXcUdeqcpck_ceUVa^XftgnfgdP[q (3576) in=40b,setxattr hehZsP\\ajqhe^nfrZjQmsnfWoqq\\e\\NV^rOaZYtj (1787) in=40b,setxattr a]Smbg`^k`emWabQRYcpr_lQj`eh`[r\\riqa_sfd (1015) in=40b,setxattr eqsaZkZ[`t^R^grhYOVrVsspVQQp_Zo\\loUV[\\U[ (982) in=40b,setxattr Rn`k`cSYQj`XOZdn_mSPrqQtoUpWTk[^[t (1647) in=34b,setxattr ]TtVabQgUTabfmqae[UVe`cNsqqYfdh]\\sVoeXQr (383) in=40b,setxattr iRn][N_NY\\ZhmYdQ[[ptTP_NW^Z]_roZnOpcmR_Y (19016) in=40b,setxattr T_b_WhnZnkaaYkNmmOUnaZa`dhRhQSshYQVQnVql (13472) in=40b,setxattr \\hZXh`dT^W[c\\XodZZ_d\\QO\\\\[PVWthO_WVOPSQd (1681) in=40b,setxattr ctSOqNY_dUqTOjgXYjqgslgNRRPoT\\\\OXPPWrtt_ (522) in=40b,setxattr f\\hTOf^sbRaYeUntSV^WhUo]qV`bSNermptfXs]Q (1645) in=40b,setxattr ohNTdfibmYs`eUTiQa^bcYW\\aarhT]YQ_haagqpf (1119) in=40b,setxattr qqUtZZ]ataUj_[pnh\\jVtU_dRpi^g[VPNeS`Y\\hk (699) in=40b,setxattr qsgedRfqdW]]cYbWOUgh^ZleaSQiteejr^afXbUj (525) in=40b,setxattr [XmPqpoSg`fWjdlmQ_qabof]_WVcPaY[]k[ardgj (594) in=40b] snapc 15=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:08:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:08:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 2.977683743, 2024-09-16T04:08:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019213674, 2024-09-16T04:08:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:47.609 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 654 2024-09-16T04:08:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:08:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:08:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534181+0000", 2024-09-16T04:08:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5830000000000001e-06 2024-09-16T04:08:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534398+0000", 2024-09-16T04:08:47.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000216663 2024-09-16T04:08:47.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534421+0000", 2024-09-16T04:08:47.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3714e-05 2024-09-16T04:08:47.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534434+0000", 2024-09-16T04:08:47.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3185e-05 2024-09-16T04:08:47.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534504+0000", 2024-09-16T04:08:47.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9665000000000005e-05 2024-09-16T04:08:47.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:08:47.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534510+0000", 2024-09-16T04:08:47.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1709999999999999e-06 2024-09-16T04:08:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.538902+0000", 2024-09-16T04:08:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043914660000000001 2024-09-16T04:08:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539044+0000", 2024-09-16T04:08:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014215000000000001 2024-09-16T04:08:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540400+0000", 2024-09-16T04:08:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013559189999999999 2024-09-16T04:08:47.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541692+0000", 2024-09-16T04:08:47.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001291964 2024-09-16T04:08:47.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:47.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553277+0000", 2024-09-16T04:08:47.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011585432999999999 2024-09-16T04:08:47.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553392+0000", 2024-09-16T04:08:47.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011476099999999999 2024-09-16T04:08:47.620 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.621 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.621 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.621 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.bs5 69/56 ECSubWrite(tid=338, reqid=osd.3.0:338, at_version=69'72, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:08:47.621 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.544515+0000", 2024-09-16T04:08:47.622 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 2.9673470640000001, 2024-09-16T04:08:47.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011654167, 2024-09-16T04:08:47.622 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:47.622 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:47.622 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:47.622 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544515+0000", 2024-09-16T04:08:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544515+0000", 2024-09-16T04:08:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.624 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544516+0000", 2024-09-16T04:08:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:08:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544539+0000", 2024-09-16T04:08:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2705e-05 2024-09-16T04:08:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:47.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544591+0000", 2024-09-16T04:08:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2037999999999999e-05 2024-09-16T04:08:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544609+0000", 2024-09-16T04:08:47.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546316+0000", 2024-09-16T04:08:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001707456 2024-09-16T04:08:47.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546327+0000", 2024-09-16T04:08:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1025e-05 2024-09-16T04:08:47.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.556143+0000", 2024-09-16T04:08:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098162240000000001 2024-09-16T04:08:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:47.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.556169+0000", 2024-09-16T04:08:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5633e-05 2024-09-16T04:08:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.631 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:47.632 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:47.632 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:47.634 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:47.635 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:47.632+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:47.738 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:47.736+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:47.749 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:47.749 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:47.749 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:47.750 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:47.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:47.839 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:47.837+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:47.861 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:47.862 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:47.862 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:47.862 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:47.862 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:47.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:47.942 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:47.940+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:48.016 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:48.016 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:48.016 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:48.016 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:48.016 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.016 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:48.016 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 38.871667236999997, 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.017 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:48.018 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:48.019 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:48.020 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.020 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.020 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.020 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:48.020 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:48.020 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.020 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.020 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.021 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:296 3.9s0 3:9fd10cc9:::smithi01639990-50:head [delete] snapc 6=[6,5] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e38)", 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 33.908787347000001, 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010626313, 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 296 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.022 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104573+0000", 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104575+0000", 2024-09-16T04:08:48.023 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1349999999999999e-06 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104586+0000", 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0603e-05 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104606+0000", 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9840999999999999e-05 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.024 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104615+0000", 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0189999999999995e-06 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104674+0000", 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9534999999999998e-05 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.104680+0000", 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9320000000000001e-06 2024-09-16T04:08:48.025 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109134+0000", 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044534120000000003 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109195+0000", 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0983000000000002e-05 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.026 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.109711+0000", 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00051621600000000001 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.110753+0000", 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001042533 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115115+0000", 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043613030000000004 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.027 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:14.115200+0000", 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4801000000000003e-05 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.553611628999999, 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:08:48.028 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.029 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:08:48.030 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.031 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:08:48.032 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.423069470000002, 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.033 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.034 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:08:48.035 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.036 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.037 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.342146917000001, 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.038 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:48.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:08:48.040 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.041 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.181742859, 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:48.042 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:08:48.043 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.044 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:423 3.9s0 3:94de6fe1:::smithi01639990-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.505744507999999, 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010412819, 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.046 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 423 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.047 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507616+0000", 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507617+0000", 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6599999999999996e-07 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507621+0000", 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9609999999999999e-06 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.048 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507628+0000", 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3139999999999998e-06 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507637+0000", 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1840000000000005e-06 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510428+0000", 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027909060000000001 2024-09-16T04:08:48.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.510505+0000", 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6679000000000002e-05 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.514954+0000", 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044494410000000002 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.515004+0000", 2024-09-16T04:08:48.050 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0244999999999998e-05 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518029+0000", 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030245229999999999 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:48.051 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.505638021999999, 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.053 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.054 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.055 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:08:48.056 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.483924996999999, 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.057 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:08:48.059 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.060 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.061 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:48.062 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.483872770000001, 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.063 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.064 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.065 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:08:48.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 67/56 ECSubWrite(tid=320, reqid=osd.3.0:320, at_version=67'62, trim_to=0'0, roll_forward_to=66'60))", 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.4808117960000002, 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01263539, 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6899999999999999e-07 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532569+0000", 2024-09-16T04:08:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0305999999999998e-05 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532682+0000", 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112361 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532692+0000", 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0832000000000001e-05 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532753+0000", 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0568e-05 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532763+0000", 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6010000000000001e-06 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545152+0000", 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012389908 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.071 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545184+0000", 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1644999999999999e-05 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 67/56 ECSubWrite(tid=321, reqid=osd.3.0:321, at_version=67'92, trim_to=0'0, roll_forward_to=66'90))", 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.532571+0000", 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.4807891499999997, 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012633584, 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.072 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532571+0000", 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532571+0000", 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532572+0000", 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.48e-07 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532583+0000", 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1116e-05 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532796+0000", 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000213406 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532803+0000", 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5880000000000003e-06 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532860+0000", 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7175000000000001e-05 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532868+0000", 2024-09-16T04:08:48.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5329999999999994e-06 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545182+0000", 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012313945 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545205+0000", 2024-09-16T04:08:48.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2572999999999999e-05 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.12s3 67/56 ECSubWrite(tid=130, reqid=osd.0.0:130, at_version=67'80, trim_to=0'0, roll_forward_to=65'78))", 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.532611+0000", 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.4807497820000002, 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012566290000000001, 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.077 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532611+0000", 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532611+0000", 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532611+0000", 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3300000000000001e-07 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532626+0000", 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5356e-05 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532929+0000", 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030242099999999998 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532935+0000", 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2450000000000003e-06 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532981+0000", 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6150999999999999e-05 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532990+0000", 2024-09-16T04:08:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7479999999999999e-06 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545137+0000", 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01214763 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545177+0000", 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9606e-05 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.11s6 68/56 ECSubWrite(tid=17, reqid=osd.10.0:17, at_version=68'19, trim_to=0'0, roll_forward_to=64'17))", 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.4741868480000004, 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014242092, 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8500000000000001e-07 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539199+0000", 2024-09-16T04:08:48.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4397999999999999e-05 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539248+0000", 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9033000000000003e-05 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539258+0000", 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8800000000000003e-06 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539858+0000", 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060046199999999996 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539867+0000", 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.363e-06 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553388+0000", 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013520249 2024-09-16T04:08:48.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553416+0000", 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8121999999999999e-05 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 68/56 ECSubWrite(tid=327, reqid=osd.3.0:327, at_version=68'66, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.47389539, 2024-09-16T04:08:48.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012797307000000001, 2024-09-16T04:08:48.087 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.087 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.087 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.087 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539466+0000", 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4000000000000001e-07 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539484+0000", 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8391999999999999e-05 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539516+0000", 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1439999999999997e-05 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539524+0000", 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4570000000000005e-06 2024-09-16T04:08:48.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542244+0000", 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027199070000000001 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542254+0000", 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8500000000000006e-06 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552231+0000", 2024-09-16T04:08:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0099774170000000006 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552262+0000", 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1204000000000001e-05 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=64, reqid=osd.4.0:64, at_version=68'43, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.4730728439999998, 2024-09-16T04:08:48.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014089379000000001, 2024-09-16T04:08:48.092 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.092 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.092 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.092 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:48.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1500000000000001e-07 2024-09-16T04:08:48.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540308+0000", 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0585999999999999e-05 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540337+0000", 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8560000000000001e-05 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540345+0000", 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6299999999999998e-06 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540400+0000", 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4818000000000003e-05 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540411+0000", 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1009000000000001e-05 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554349+0000", 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013938394 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554377+0000", 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8167e-05 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=66, reqid=osd.4.0:66, at_version=68'45, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.471231553, 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01391278, 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542130+0000", 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6600000000000005e-07 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542154+0000", 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3862999999999998e-05 2024-09-16T04:08:48.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542183+0000", 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9108999999999999e-05 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542192+0000", 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549962+0000", 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007769977 2024-09-16T04:08:48.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549972+0000", 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0074e-05 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556017+0000", 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060444319999999998 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556042+0000", 2024-09-16T04:08:48.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5012999999999999e-05 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s7 68/56 ECSubWrite(tid=197, reqid=osd.7.0:197, at_version=68'24, trim_to=0'0, roll_forward_to=64'22))", 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.4708881180000004, 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013604978, 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.101 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542473+0000", 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5099999999999999e-07 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542491+0000", 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8417e-05 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542561+0000", 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9744000000000002e-05 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542568+0000", 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.289e-06 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546547+0000", 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039792350000000002 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546557+0000", 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8870000000000004e-06 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556052+0000", 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094947169999999997 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556077+0000", 2024-09-16T04:08:48.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5537999999999998e-05 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 68/56 ECSubWrite(tid=329, reqid=osd.3.0:329, at_version=68'68, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542798+0000", 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.470562277, 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011606005000000001, 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542798+0000", 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542798+0000", 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542799+0000", 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542822+0000", 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2697999999999999e-05 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542858+0000", 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6322000000000002e-05 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542868+0000", 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5959999999999993e-06 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546596+0000", 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003727981 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546605+0000", 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1039999999999996e-06 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554384+0000", 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077793009999999998 2024-09-16T04:08:48.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554404+0000", 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.014e-05 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 69/56 ECSubWrite(tid=338, reqid=osd.3.0:338, at_version=69'72, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.544371+0000", 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.468989986, 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011923731999999999, 2024-09-16T04:08:48.110 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544371+0000", 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544371+0000", 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544371+0000", 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4899999999999995e-07 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544389+0000", 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7895999999999999e-05 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544413+0000", 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4022000000000001e-05 2024-09-16T04:08:48.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544420+0000", 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6950000000000003e-06 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546069+0000", 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016493059999999999 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546078+0000", 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2329999999999995e-06 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556258+0000", 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010179532 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556294+0000", 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6399000000000002e-05 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.114 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.115 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.115 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.115 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.115 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:48.117 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:48.117 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:48.114+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:48.217 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:48.216+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:48.220 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:48.220 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:48.220 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:48.220 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:48.229 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:48.319 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:48.317+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:48.331 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:48.332 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:48.332 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:48.332 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:48.332 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:48.340 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:48.420 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:48.418+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:48.439 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-16T04:08:48.442 INFO:tasks.rados.rados.0.smithi016.stdout:289: done (0 left) 2024-09-16T04:08:48.443 INFO:tasks.rados.rados.0.smithi016.stdout:290: read oid 43 snap 22 2024-09-16T04:08:48.443 INFO:tasks.rados.rados.0.smithi016.stdout:290: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:08:48.443 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 27 current snap is 23 2024-09-16T04:08:48.443 INFO:tasks.rados.rados.0.smithi016.stdout:291: seq_num 127 ranges {0=589824,589824=753664,1343488=475136,1818624=327680} 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 41.348846393999999, 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.446 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.447 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:48.448 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.449 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.450 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 42/27 ECSubWrite(tid=155, reqid=osd.3.0:155, at_version=42'28, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 31.003513588000001, 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073085160000000001, 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:48.451 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439640+0000", 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1299999999999999e-07 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.452 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439663+0000", 2024-09-16T04:08:48.453 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3030000000000001e-05 2024-09-16T04:08:48.453 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.453 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.453 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.453 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439705+0000", 2024-09-16T04:08:48.453 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.1078999999999998e-05 2024-09-16T04:08:48.453 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.453 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.454 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.454 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.439714+0000", 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5470000000000002e-06 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441461+0000", 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017466350000000001 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.441475+0000", 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4404e-05 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.455 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446911+0000", 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0054361430000000001 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.446949+0000", 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7465000000000001e-05 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.456 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.013329423999998, 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.457 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.458 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.459 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.459 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:08:48.459 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:08:48.459 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.459 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.459 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.459 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:08:48.460 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:08:48.460 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.460 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.460 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.460 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:08:48.460 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.553379197000002, 2024-09-16T04:08:48.461 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:08:48.462 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.463 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.463 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.463 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.463 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.463 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:08:48.464 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.466 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.467 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.229866592, 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.468 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.469 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:08:48.470 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.471 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.972082734000001, 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078488229999999996, 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:48.472 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471072+0000", 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0279999999999999e-06 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471418+0000", 2024-09-16T04:08:48.473 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034541800000000001 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471457+0000", 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471492+0000", 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.474 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471798+0000", 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030615599999999997 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471812+0000", 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.448e-05 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478881+0000", 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070685549999999998 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.475 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478920+0000", 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9072000000000001e-05 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.002968266, 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084259260000000002, 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.476 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.477 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.91e-07 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440211+0000", 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4978999999999999e-05 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.478 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440257+0000", 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6569000000000001e-05 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440268+0000", 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1056000000000001e-05 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440348+0000", 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9795000000000004e-05 2024-09-16T04:08:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440364+0000", 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5996999999999999e-05 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448570+0000", 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082059590000000005 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.480 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448611+0000", 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.138e-05 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.bs7 57/56 [PushOp(3:d683cb60:::smithi01639990-49:head, version: 53'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d683cb60:::smithi01639990-49:head@53'38, size: 2310144, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.882386094999999, 2024-09-16T04:08:48.481 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.043702234999999999, 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.482 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560769+0000", 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0049999999999999e-06 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.561301+0000", 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053276500000000004 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.483 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603955+0000", 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042653605999999997 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603967+0000", 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2065000000000001e-05 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604063+0000", 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5599000000000004e-05 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.484 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604470+0000", 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040719500000000001 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=252, reqid=osd.3.0:252, at_version=57'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.533659954999999, 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27497091299999998, 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.485 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.486 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909517+0000", 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2886000000000001e-05 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183847+0000", 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27432981400000001 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.487 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183853+0000", 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2929999999999997e-06 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183902+0000", 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9008999999999998e-05 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183911+0000", 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:08:48.488 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184435+0000", 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052493699999999995 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184465+0000", 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9292e-05 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.489 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=253, reqid=osd.3.0:253, at_version=57'67, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.533031923999999, 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50851210899999999, 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.490 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910123+0000", 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910136+0000", 2024-09-16T04:08:48.491 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.331e-05 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416910+0000", 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50677369000000005 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416919+0000", 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8290000000000003e-06 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.492 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416975+0000", 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5983999999999998e-05 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416985+0000", 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418599+0000", 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001614515 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.493 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418634+0000", 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4588999999999997e-05 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 64/56 ECSubWrite(tid=285, reqid=client.4440.0:535, at_version=0'0, trim_to=0'0, roll_forward_to=64'72))", 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.9015848549999994, 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012095003, 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.494 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.495 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541570+0000", 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.173e-06 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541582+0000", 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2106999999999999e-05 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552877+0000", 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011294692 2024-09-16T04:08:48.496 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552889+0000", 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1875000000000001e-05 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552958+0000", 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9510000000000004e-05 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.497 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552967+0000", 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6240000000000008e-06 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553631+0000", 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066445299999999996 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553664+0000", 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2568999999999999e-05 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.498 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 68/46 ECSubWrite(tid=65, reqid=osd.4.0:65, at_version=68'22, trim_to=0'0, roll_forward_to=67'20))", 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9018585970000004, 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010787494999999999, 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:48.499 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.500 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541313+0000", 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7431e-05 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541369+0000", 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6311999999999999e-05 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541376+0000", 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5729999999999996e-06 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.501 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544486+0000", 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031104150000000001 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544495+0000", 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0960000000000007e-06 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.502 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552010+0000", 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075148200000000002 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552083+0000", 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2661000000000001e-05 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.503 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 68/56 ECSubWrite(tid=329, reqid=osd.3.0:329, at_version=68'68, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.9004048769999997, 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093665009999999993, 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:48.504 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542750+0000", 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.23e-07 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.505 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542770+0000", 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0679999999999999e-05 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542842+0000", 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1391999999999997e-05 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542850+0000", 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2320000000000001e-06 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.506 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545061+0000", 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022114220000000002 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545070+0000", 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6780000000000006e-06 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552088+0000", 2024-09-16T04:08:48.507 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070176939999999997 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552115+0000", 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7679999999999999e-05 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds2 69/56 ECSubWrite(tid=201, reqid=client.4440.0:654, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:48.508 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.9025670130000001, 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090194439999999997, 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.509 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1400000000000001e-07 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540777+0000", 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000189886 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.510 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540803+0000", 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6466e-05 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540811+0000", 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0730000000000005e-06 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540879+0000", 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8003999999999995e-05 2024-09-16T04:08:48.511 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540889+0000", 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8460000000000003e-06 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.549571+0000", 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086822059999999996 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.512 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.549606+0000", 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4749000000000002e-05 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=336, reqid=osd.3.0:336, at_version=69'70, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.899460833, 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077691619999999996, 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.513 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.514 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.05e-07 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543708+0000", 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4431e-05 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543897+0000", 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018918 2024-09-16T04:08:48.515 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543906+0000", 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3349999999999996e-06 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543954+0000", 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7793999999999997e-05 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.516 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543963+0000", 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4259999999999992e-06 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551434+0000", 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0074706800000000004 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551462+0000", 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8110999999999999e-05 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.517 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.518 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.518 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.518 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=338, reqid=osd.3.0:338, at_version=69'72, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:08:48.518 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:08:48.518 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.8987666490000001, 2024-09-16T04:08:48.518 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010064081000000001, 2024-09-16T04:08:48.518 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.518 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.518 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.519 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544388+0000", 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9800000000000003e-07 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544409+0000", 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0993e-05 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544442+0000", 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2937e-05 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.520 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544449+0000", 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.588e-06 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547347+0000", 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028976599999999998 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547357+0000", 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6479999999999998e-06 2024-09-16T04:08:48.521 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554419+0000", 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070620819999999999 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554451+0000", 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2475000000000003e-05 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.522 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s6 69/56 ECSubWrite(tid=202, reqid=osd.7.0:202, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.8974298539999999, 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010752746000000001, 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.523 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545725+0000", 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4199999999999995e-07 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545750+0000", 2024-09-16T04:08:48.524 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4915000000000001e-05 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546880+0000", 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011307870000000001 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546891+0000", 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0825000000000001e-05 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.525 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551495+0000", 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004604131 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551504+0000", 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6230000000000003e-06 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556450+0000", 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049460210000000001 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.526 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.527 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.527 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556477+0000", 2024-09-16T04:08:48.527 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6801999999999999e-05 2024-09-16T04:08:48.527 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.527 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.527 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.527 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds2 69/56 ECSubWrite(tid=203, reqid=osd.7.0:203, at_version=69'28, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.8974008179999999, 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078627160000000005, 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:48.528 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.91e-07 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.529 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545769+0000", 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5562000000000001e-05 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546923+0000", 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001154457 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546936+0000", 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3183000000000001e-05 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.530 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547734+0000", 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00079775500000000004 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547744+0000", 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5489999999999995e-06 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.553577+0000", 2024-09-16T04:08:48.531 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058334809999999997 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.553616+0000", 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8438e-05 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 69/56 ECSubWrite(tid=340, reqid=osd.3.0:340, at_version=69'98, trim_to=0'0, roll_forward_to=68'96))", 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.554920+0000", 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.8882339109999999, 2024-09-16T04:08:48.532 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075442290000000004, 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554920+0000", 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554920+0000", 2024-09-16T04:08:48.533 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554921+0000", 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9800000000000003e-07 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554941+0000", 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0653e-05 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.534 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554976+0000", 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4981000000000003e-05 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554987+0000", 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0336000000000001e-05 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.555043+0000", 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6787999999999999e-05 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.535 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.555054+0000", 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0205e-05 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.562428+0000", 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073745640000000001 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.562464+0000", 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6004000000000003e-05 2024-09-16T04:08:48.536 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.15s7 69/48 ECSubWrite(tid=117, reqid=osd.8.0:117, at_version=69'59, trim_to=0'0, roll_forward_to=68'57))", 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.557667+0000", 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.88548728, 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076099649999999998, 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.537 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557667+0000", 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557667+0000", 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557667+0000", 2024-09-16T04:08:48.538 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3099999999999997e-07 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557688+0000", 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0990000000000001e-05 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557718+0000", 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9444000000000001e-05 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.539 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557726+0000", 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7290000000000006e-06 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557845+0000", 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011856 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557856+0000", 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0832999999999999e-05 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.540 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.564509+0000", 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0066528170000000001 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.565276+0000", 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00076796099999999995 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.541 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:48.543 INFO:tasks.rados.rados.0.smithi016.stdout:291: writing smithi01639990-27 from 0 to 589824 tid 1 2024-09-16T04:08:48.543 INFO:tasks.rados.rados.0.smithi016.stdout:291: writing smithi01639990-27 from 589824 to 1343488 tid 2 2024-09-16T04:08:48.544 INFO:tasks.rados.rados.0.smithi016.stdout:291: writing smithi01639990-27 from 1343488 to 1818624 tid 3 2024-09-16T04:08:48.544 INFO:tasks.rados.rados.0.smithi016.stdout:291: writing smithi01639990-27 from 1818624 to 2146304 tid 4 2024-09-16T04:08:48.544 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 47 current snap is 23 2024-09-16T04:08:48.544 INFO:tasks.rados.rados.0.smithi016.stdout:292: seq_num 128 ranges {3391488=606208,3997696=507904,4505600=573440,5079040=475136} 2024-09-16T04:08:48.544 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:48.546 INFO:tasks.rados.rados.0.smithi016.stdout:292: writing smithi01639990-47 from 3391488 to 3997696 tid 1 2024-09-16T04:08:48.546 INFO:tasks.rados.rados.0.smithi016.stdout:292: writing smithi01639990-47 from 3997696 to 4505600 tid 2 2024-09-16T04:08:48.546 INFO:tasks.rados.rados.0.smithi016.stdout:292: writing smithi01639990-47 from 4505600 to 5079040 tid 3 2024-09-16T04:08:48.546 INFO:tasks.rados.rados.0.smithi016.stdout:292: writing smithi01639990-47 from 5079040 to 5554176 tid 4 2024-09-16T04:08:48.546 INFO:tasks.rados.rados.0.smithi016.stdout:293: rmattr oid 26 current snap is 23 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:290: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: finishing write tid 1 to smithi01639990-27 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: finishing write tid 2 to smithi01639990-27 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: finishing write tid 3 to smithi01639990-27 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: finishing write tid 4 to smithi01639990-27 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: finishing write tid 5 to smithi01639990-27 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: finishing write tid 6 to smithi01639990-27 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: oid 27 updating version 0 to 28 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: oid 27 version 28 is already newer than 26 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: oid 27 version 28 is already newer than 27 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: oid 27 version 28 is already newer than 24 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:291: oid 27 version 28 is already newer than 25 2024-09-16T04:08:48.547 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 27 v 28 (ObjNum 127 snap 23 seq_num 127) dirty exists 2024-09-16T04:08:48.548 INFO:tasks.rados.rados.0.smithi016.stdout:291: left oid 27 (ObjNum 127 snap 23 seq_num 127) 2024-09-16T04:08:48.548 INFO:tasks.rados.rados.0.smithi016.stdout:292: finishing write tid 1 to smithi01639990-47 2024-09-16T04:08:48.548 INFO:tasks.rados.rados.0.smithi016.stdout:290: done (3 left) 2024-09-16T04:08:48.548 INFO:tasks.rados.rados.0.smithi016.stdout:291: done (2 left) 2024-09-16T04:08:48.548 INFO:tasks.rados.rados.0.smithi016.stdout:294: copy_from oid 1 from oid 18 current snap is 23 2024-09-16T04:08:48.548 INFO:tasks.rados.rados.0.smithi016.stdout:292: finishing write tid 2 to smithi01639990-47 2024-09-16T04:08:48.548 INFO:tasks.rados.rados.0.smithi016.stdout:292: finishing write tid 3 to smithi01639990-47 2024-09-16T04:08:48.548 INFO:tasks.rados.rados.0.smithi016.stdout:292: finishing write tid 4 to smithi01639990-47 2024-09-16T04:08:48.548 INFO:tasks.rados.rados.0.smithi016.stdout:295: snap_remove snap 17 2024-09-16T04:08:48.549 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:48.541+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:48.644 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:48.642+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:48.651 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:48.651 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:48.651 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:48.660 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:48.745 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:48.743+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:48.765 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:48.765 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:48.765 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:48.765 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:48.765 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:48.771 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:48.846 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:48.844+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:48.876 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:48.876 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:48.876 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:48.876 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:48.876 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.876 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 48/46 ECSubWrite(tid=190, reqid=osd.3.0:190, at_version=48'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:48.876 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:48.876 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.407707996999999, 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088361580000000002, 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:48.877 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465645+0000", 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4399999999999999e-07 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465670+0000", 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4875000000000002e-05 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.878 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465791+0000", 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012066900000000001 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465800+0000", 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4539999999999996e-06 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465872+0000", 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2127999999999998e-05 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.879 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465881+0000", 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3129999999999999e-06 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474449+0000", 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085683760000000008 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474480+0000", 2024-09-16T04:08:48.880 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1498999999999998e-05 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 52/46 ECSubWrite(tid=210, reqid=client.4440.0:434, at_version=52'63, trim_to=0'0, roll_forward_to=50'60))", 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.355585471000001, 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83621405000000004, 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.881 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.882 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8200000000000001e-07 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517790+0000", 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2540000000000001e-05 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352789+0000", 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83499984500000002 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.883 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352804+0000", 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4569999999999999e-05 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352928+0000", 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012396799999999999 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352943+0000", 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:48.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353943+0000", 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099977500000000006 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353981+0000", 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7700999999999997e-05 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.885 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=217, reqid=client.4440.0:449, at_version=53'58, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.362075435000001, 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0096255799999999999, 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.886 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511278+0000", 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.068e-06 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511623+0000", 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034493499999999999 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.887 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515041+0000", 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034178889999999999 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515048+0000", 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3479999999999998e-06 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515115+0000", 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6469000000000003e-05 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.888 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515126+0000", 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1187e-05 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520872+0000", 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057460280000000002 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520902+0000", 2024-09-16T04:08:48.889 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0655999999999999e-05 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=218, reqid=client.4440.0:450, at_version=53'59, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.358730860000001, 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089217440000000005, 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.890 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514623+0000", 2024-09-16T04:08:48.891 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2300000000000001e-06 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515021+0000", 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039790399999999998 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519384+0000", 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043635380000000001 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.892 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519391+0000", 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9709999999999998e-06 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519441+0000", 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9493999999999997e-05 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519449+0000", 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:08:48.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.523504+0000", 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040551220000000004 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.523543+0000", 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9271999999999999e-05 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.894 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=220, reqid=client.4440.0:452, at_version=53'61, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.353581918, 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51827822899999998, 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.895 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519771+0000", 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.61e-07 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.896 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519788+0000", 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6699999999999999e-05 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036882+0000", 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51709414300000001 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036895+0000", 2024-09-16T04:08:48.897 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3363999999999999e-05 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036964+0000", 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8734999999999998e-05 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036973+0000", 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0289999999999994e-06 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038014+0000", 2024-09-16T04:08:48.898 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010410529999999999 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038049+0000", 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4443999999999998e-05 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:526 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~622592] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:48.899 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.242785530000001, 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01502261, 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 526 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.900 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630568+0000", 2024-09-16T04:08:48.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5680000000000001e-06 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631353+0000", 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078490700000000005 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631367+0000", 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3441000000000001e-05 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631375+0000", 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8809999999999996e-06 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631465+0000", 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0673000000000004e-05 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631565+0000", 2024-09-16T04:08:48.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9753000000000005e-05 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.632641+0000", 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001076097 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634418+0000", 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017769680000000001 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.904 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645537+0000", 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011119393999999999 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645589+0000", 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1928000000000002e-05 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:527 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:48.905 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.241948456999999, 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015053211, 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 527 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.906 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631405+0000", 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.08e-06 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631411+0000", 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9009999999999999e-06 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631419+0000", 2024-09-16T04:08:48.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:08:48.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631426+0000", 2024-09-16T04:08:48.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4999999999999996e-06 2024-09-16T04:08:48.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633558+0000", 2024-09-16T04:08:48.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002131796 2024-09-16T04:08:48.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633663+0000", 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105833 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634106+0000", 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00044296899999999998 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.636380+0000", 2024-09-16T04:08:48.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022740849999999999 2024-09-16T04:08:48.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.912 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646383+0000", 2024-09-16T04:08:48.912 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01000238 2024-09-16T04:08:48.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.912 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646457+0000", 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4258000000000006e-05 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:528 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.241873876, 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65958352099999995, 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.913 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 528 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:48.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631479+0000", 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9599999999999998e-07 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631483+0000", 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8160000000000004e-06 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.917 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.917 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631494+0000", 2024-09-16T04:08:48.917 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0407999999999999e-05 2024-09-16T04:08:48.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.918 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.918 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631501+0000", 2024-09-16T04:08:48.918 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2269999999999996e-06 2024-09-16T04:08:48.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.918 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.918 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634655+0000", 2024-09-16T04:08:48.918 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031544339999999998 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634732+0000", 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6495999999999994e-05 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646864+0000", 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012131807999999999 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646965+0000", 2024-09-16T04:08:48.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010173 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.291062+0000", 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64409670600000002 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.920 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 67/56 ECSubWrite(tid=321, reqid=osd.3.0:321, at_version=67'92, trim_to=0'0, roll_forward_to=66'90))", 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.533170+0000", 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.3401819369999997, 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086084990000000004, 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533170+0000", 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533170+0000", 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533171+0000", 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.05e-07 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533221+0000", 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0498999999999999e-05 2024-09-16T04:08:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533284+0000", 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2853000000000003e-05 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533293+0000", 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2529999999999993e-06 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533356+0000", 2024-09-16T04:08:48.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3107000000000004e-05 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533365+0000", 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9390000000000003e-06 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541744+0000", 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083788960000000003 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541779+0000", 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4746999999999998e-05 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s7 67/46 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=67'20, trim_to=0'0, roll_forward_to=66'18))", 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.3397385330000002, 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090747759999999997, 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.925 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:48.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8199999999999999e-07 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533635+0000", 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1378000000000001e-05 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533659+0000", 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.374e-05 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533667+0000", 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2230000000000007e-06 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533785+0000", 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117607 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533795+0000", 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0180000000000001e-05 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.928 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542641+0000", 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088456149999999994 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542689+0000", 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7851e-05 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s2 68/56 ECSubWrite(tid=134, reqid=osd.0.0:134, at_version=68'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:48.929 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.3313175560000001, 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010826218, 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:48.930 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8599999999999998e-07 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542054+0000", 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8423999999999998e-05 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542081+0000", 2024-09-16T04:08:48.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7806000000000001e-05 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542089+0000", 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5789999999999998e-06 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542147+0000", 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7978e-05 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542157+0000", 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552766+0000", 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010609143999999999 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552861+0000", 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4841999999999998e-05 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.933 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 68/56 ECSubWrite(tid=330, reqid=osd.3.0:330, at_version=68'75, trim_to=0'0, roll_forward_to=64'73))", 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.3302626000000002, 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094449849999999995, 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:48.934 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6000000000000003e-07 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.935 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543110+0000", 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9695e-05 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543139+0000", 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8467999999999999e-05 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543148+0000", 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5349999999999993e-06 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543200+0000", 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1536999999999998e-05 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543206+0000", 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9379999999999997e-06 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552495+0000", 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.009289538 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552535+0000", 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9613999999999997e-05 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:695 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~589824] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.42432588399999999, 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014512838, 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.938 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 695 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.939 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449028+0000", 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.917e-06 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450009+0000", 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000980367 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450023+0000", 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.416e-05 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450032+0000", 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5820000000000008e-06 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450094+0000", 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1698000000000005e-05 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450210+0000", 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116104 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451973+0000", 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001762958 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463392+0000", 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011419024999999999 2024-09-16T04:08:48.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463450+0000", 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8251999999999997e-05 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463539+0000", 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8775000000000003e-05 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s7 73/56 ECSubWrite(tid=129, reqid=osd.8.0:129, at_version=73'80, trim_to=0'0, roll_forward_to=71'78))", 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.451542+0000", 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.42181074699999999, 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086628820000000002, 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451542+0000", 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451542+0000", 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451542+0000", 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.92e-07 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451563+0000", 2024-09-16T04:08:48.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1348e-05 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451598+0000", 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4958e-05 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451607+0000", 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5369999999999997e-06 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451663+0000", 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6773e-05 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451673+0000", 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5570000000000001e-06 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460177+0000", 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085042820000000002 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.947 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460204+0000", 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7234999999999999e-05 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:696 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~753664] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.42119345000000002, 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013764281, 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.948 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 696 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452161+0000", 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2579999999999998e-06 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453157+0000", 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099543299999999999 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453172+0000", 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5917e-05 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453182+0000", 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3959999999999995e-06 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455292+0000", 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021103379999999998 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455406+0000", 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000114203 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456783+0000", 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001376377 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463544+0000", 2024-09-16T04:08:48.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067608640000000001 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465825+0000", 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022816350000000002 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465923+0000", 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7860000000000002e-05 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:697 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~475136] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.456443+0000", 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.41690882200000001, 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012866471000000001, 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 697 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.954 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456443+0000", 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456443+0000", 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456445+0000", 2024-09-16T04:08:48.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6220000000000001e-06 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457074+0000", 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000629248 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457087+0000", 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2906e-05 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457096+0000", 2024-09-16T04:08:48.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9449999999999999e-06 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457956+0000", 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00086007 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458062+0000", 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105559 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.459157+0000", 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00109566 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463617+0000", 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044591639999999998 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469249+0000", 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056324160000000003 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469310+0000", 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0881000000000002e-05 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:698 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~327680] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.458939+0000", 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.41441361900000001, 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010478334000000001, 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.959 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 698 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458939+0000", 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.960 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458939+0000", 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458940+0000", 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5770000000000001e-06 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460359+0000", 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001418334 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460372+0000", 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3025e-05 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460379+0000", 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4680000000000003e-06 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460452+0000", 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.339e-05 2024-09-16T04:08:48.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460661+0000", 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00020814299999999999 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.461550+0000", 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00088961800000000001 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465933+0000", 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043829330000000003 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469379+0000", 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034461069999999999 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469417+0000", 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7738999999999999e-05 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:699 3.11s0 3:8bfd968c:::smithi01639990-27:head [setxattr _header (57) in=7b] snapc 18=[18,16,15,13] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.471289+0000", 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.402063161, 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012440791, 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 699 2024-09-16T04:08:48.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.969 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471289+0000", 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471289+0000", 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471290+0000", 2024-09-16T04:08:48.970 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1800000000000005e-07 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471292+0000", 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6180000000000002e-06 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471297+0000", 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7419999999999997e-06 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.971 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471302+0000", 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9320000000000004e-06 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471382+0000", 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9864000000000006e-05 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471444+0000", 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2249999999999995e-05 2024-09-16T04:08:48.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471785+0000", 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000340267 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472441+0000", 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00065623500000000002 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:48.973 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.483660+0000", 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011219632 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.483730+0000", 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9533e-05 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:700 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:48.974 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.40204294200000001, 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014504263, 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 700 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:08:48.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471310+0000", 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.08e-07 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471312+0000", 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0540000000000002e-06 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471318+0000", 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5080000000000002e-06 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471322+0000", 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3629999999999998e-06 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472045+0000", 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00072327100000000005 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472133+0000", 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8528000000000003e-05 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484130+0000", 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011996163000000001 2024-09-16T04:08:48.978 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484213+0000", 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3800000000000004e-05 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.485814+0000", 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016001679999999999 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.979 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.34271804700000003, 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095992542, 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.980 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530636+0000", 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.522e-06 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:48.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530651+0000", 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4980999999999999e-05 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625860+0000", 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095209590999999996 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625873+0000", 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2413e-05 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:48.982 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625978+0000", 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010492099999999999 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625994+0000", 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6058000000000001e-05 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626583+0000", 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058931700000000001 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:48.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:48.984 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626627+0000", 2024-09-16T04:08:48.984 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3739000000000003e-05 2024-09-16T04:08:48.984 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.984 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.984 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.984 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:48.984 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:48.984 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:48.986 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:48.986 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:48.984+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:49.086 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:49.085+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:49.128 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:49.129 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:49.129 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:49.129 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:49.137 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:49.187 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:49.186+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:49.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:49.239 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:49.240 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:49.240 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:49.240 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:49.240 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:49.249 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:49.289 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:49.287+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:49.354 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:124 1.0 e22/19 1:30306672:devicehealth::main.db.0000000000000000:head v 22'86, mlcod=21'84)", 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 56.921242435000003, 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020095396000000001, 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:49.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:49.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430782+0000", 2024-09-16T04:08:49.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9699999999999996e-07 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430789+0000", 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4660000000000002e-06 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450028+0000", 2024-09-16T04:08:49.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019238816999999998 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450042+0000", 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4487e-05 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450169+0000", 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126787 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450188+0000", 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450816+0000", 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062741399999999997 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450877+0000", 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0782999999999997e-05 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.360 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.361 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.361 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:197 3.es0 3:7f241f48:::smithi01639990-30:head [copy-from ver 23] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:49.361 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:49.361 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 41.142033087000002, 2024-09-16T04:08:49.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.023242381999999999, 2024-09-16T04:08:49.361 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.361 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.362 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:49.362 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.362 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.362 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.362 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 197 2024-09-16T04:08:49.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.362 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:49.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:49.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:49.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6499999999999999e-07 2024-09-16T04:08:49.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209995+0000", 2024-09-16T04:08:49.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.388e-06 2024-09-16T04:08:49.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210002+0000", 2024-09-16T04:08:49.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5209999999999997e-06 2024-09-16T04:08:49.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.367 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210008+0000", 2024-09-16T04:08:49.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8780000000000003e-06 2024-09-16T04:08:49.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:49.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8421999999999999e-05 2024-09-16T04:08:49.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210099+0000", 2024-09-16T04:08:49.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2469999999999997e-05 2024-09-16T04:08:49.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:49.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227450+0000", 2024-09-16T04:08:49.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017351235 2024-09-16T04:08:49.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:49.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232352+0000", 2024-09-16T04:08:49.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049017239999999997 2024-09-16T04:08:49.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:49.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233168+0000", 2024-09-16T04:08:49.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081616600000000001 2024-09-16T04:08:49.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233233+0000", 2024-09-16T04:08:49.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5012999999999996e-05 2024-09-16T04:08:49.373 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.373 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.373 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.374 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:198 3.es0 3:7f241f48:::smithi01639990-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:49.374 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:49.374 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 41.142002777999998, 2024-09-16T04:08:49.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019673758, 2024-09-16T04:08:49.375 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.375 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.375 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:49.375 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.375 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.375 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.376 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 198 2024-09-16T04:08:49.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.376 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.376 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:49.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3099999999999998e-07 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210024+0000", 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2620000000000001e-06 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210030+0000", 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9020000000000002e-06 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210037+0000", 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1480000000000003e-06 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210227+0000", 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00019055499999999999 2024-09-16T04:08:49.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210269+0000", 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2246999999999997e-05 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229480+0000", 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019210896000000002 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229587+0000", 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107026 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229695+0000", 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107291 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.385 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:213 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 40.235924273000002, 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53275775299999995, 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 213 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.386 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116100+0000", 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116103+0000", 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.174e-06 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116109+0000", 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4330000000000003e-06 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116114+0000", 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1399999999999999e-06 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119298+0000", 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031846729999999998 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119365+0000", 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6484999999999998e-05 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119675+0000", 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00031021999999999999 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122365+0000", 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002689851 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648785+0000", 2024-09-16T04:08:49.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52641971300000001 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648857+0000", 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2408000000000002e-05 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:214 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 40.235901597000002, 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53416417900000002, 2024-09-16T04:08:49.391 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 214 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2100000000000002e-07 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116125+0000", 2024-09-16T04:08:49.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2649999999999999e-06 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116132+0000", 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5190000000000003e-06 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116137+0000", 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9869999999999999e-06 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120036+0000", 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038984979999999998 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120118+0000", 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2312999999999996e-05 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649191+0000", 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52907320199999996 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649279+0000", 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7585999999999998e-05 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.650286+0000", 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010073879999999999 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.396 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "replica_scrub(pg: 3.4s1,from:0'0,to:37'17,epoch:47/46,start:3:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.494100060000001, 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16799329900000001, 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:49.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:49.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9400000000000005e-07 2024-09-16T04:08:49.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857929+0000", 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4719999999999997e-06 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025793+0000", 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16786378299999999 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025803+0000", 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9629999999999993e-06 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025875+0000", 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2914e-05 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025887+0000", 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1345999999999999e-05 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025917+0000", 2024-09-16T04:08:49.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0127000000000001e-05 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.4s1,RELEASE,e:47,from:1(0),reservation_nonce:0)", 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.324955084999999, 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26135687200000002, 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.401 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2800000000000005e-07 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027075+0000", 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3859999999999998e-06 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288302+0000", 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26122710999999998 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.403 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288308+0000", 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6189999999999999e-06 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288392+0000", 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4549000000000007e-05 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288414+0000", 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1576000000000001e-05 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288425+0000", 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1704e-05 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:385 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.885952812999999, 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01911771, 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.405 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 385 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.406 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466073+0000", 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4389999999999999e-06 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466084+0000", 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1157e-05 2024-09-16T04:08:49.407 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466101+0000", 2024-09-16T04:08:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6334999999999999e-05 2024-09-16T04:08:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466111+0000", 2024-09-16T04:08:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466255+0000", 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014411 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466263+0000", 2024-09-16T04:08:49.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.858e-06 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474564+0000", 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083014019999999994 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474734+0000", 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016996599999999999 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:49.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475786+0000", 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010515909999999999 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.482124+0000", 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063382530000000003 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485117+0000", 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029928120000000001 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.411 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485189+0000", 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1868999999999999e-05 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:386 3.8s0 3:17b72cd6:::smithi01639990-47:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.885881080000001, 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019422101000000001, 2024-09-16T04:08:49.412 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 386 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8100000000000002e-07 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466147+0000", 2024-09-16T04:08:49.414 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6559999999999998e-06 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466155+0000", 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6240000000000002e-06 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466162+0000", 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4850000000000003e-06 2024-09-16T04:08:49.415 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466343+0000", 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000181077 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466349+0000", 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1229999999999997e-06 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.416 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476407+0000", 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010057885000000001 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476516+0000", 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108738 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477098+0000", 2024-09-16T04:08:49.417 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000582024 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484050+0000", 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0069524950000000004 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485519+0000", 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014691280000000001 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.418 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485565+0000", 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5185000000000002e-05 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:387 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.885852978999999, 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51761721800000005, 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.419 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 387 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.420 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6399999999999998e-07 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466173+0000", 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.396e-06 2024-09-16T04:08:49.421 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466179+0000", 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9290000000000003e-06 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466184+0000", 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9760000000000003e-06 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466397+0000", 2024-09-16T04:08:49.422 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021269400000000001 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466406+0000", 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0769999999999997e-06 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479302+0000", 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012896014000000001 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.423 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479415+0000", 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112767 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479914+0000", 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00049861199999999995 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484063+0000", 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004149158 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.424 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983716+0000", 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49965376500000003 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983788+0000", 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1465999999999996e-05 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/46 ECSubWriteReply(tid=59, last_complete=48'38, committed=1, applied=1))", 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:49.425 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.867811967000002, 2024-09-16T04:08:49.426 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50006588100000005, 2024-09-16T04:08:49.426 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.426 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.426 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:49.426 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.426 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.426 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.426 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:49.426 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2499999999999999e-07 2024-09-16T04:08:49.427 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484214+0000", 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6440000000000001e-06 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983573+0000", 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49935939200000001 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.428 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983581+0000", 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983628+0000", 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7642999999999998e-05 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.984278+0000", 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00064939900000000005 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "rep_scrubmap(3.8s0 e46 from shard 10(6))", 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.365710674999999, 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21666559099999999, 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986314+0000", 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5600000000000005e-07 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.431 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986319+0000", 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0950000000000003e-06 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202862+0000", 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21654300900000001 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202872+0000", 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0077e-05 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.432 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202931+0000", 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9568000000000003e-05 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202942+0000", 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0139999999999999e-05 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202979+0000", 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6946000000000002e-05 2024-09-16T04:08:49.433 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s3 53/48 ECSubWrite(tid=39, reqid=osd.4.0:39, at_version=53'24, trim_to=0'0, roll_forward_to=51'22))", 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.863085887, 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017845360000000001, 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.434 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2000000000000001e-07 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.435 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488964+0000", 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6418e-05 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502624+0000", 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013659439000000001 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502636+0000", 2024-09-16T04:08:49.436 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2401999999999999e-05 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502702+0000", 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6006999999999993e-05 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502712+0000", 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8870000000000004e-06 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:49.437 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506740+0000", 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040279319999999997 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506783+0000", 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3055e-05 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:446 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:49.438 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.848484228, 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67862900900000001, 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 446 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.439 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503541+0000", 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3680000000000001e-06 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.440 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503545+0000", 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.561e-06 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503553+0000", 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8939999999999993e-06 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503559+0000", 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.224e-06 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.441 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505239+0000", 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001679739 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505303+0000", 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4219000000000003e-05 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505673+0000", 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000369374 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.442 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.507082+0000", 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014090330000000001 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182055+0000", 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67497357700000005 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182168+0000", 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011302 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.443 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:447 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.848454722, 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.680085987, 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.444 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 447 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.445 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503570+0000", 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.44e-07 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503576+0000", 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4559999999999997e-06 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503602+0000", 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6264000000000001e-05 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503610+0000", 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4109999999999999e-06 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505945+0000", 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023353390000000001 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506004+0000", 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9410999999999998e-05 2024-09-16T04:08:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182603+0000", 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67659811199999997 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182695+0000", 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1951000000000001e-05 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.448 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.183655+0000", 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00096039900000000002 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.843382396999999, 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.299096274, 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.450 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508642+0000", 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5400000000000002e-07 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508645+0000", 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9249999999999999e-06 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807520+0000", 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.29887539899999999 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.451 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807534+0000", 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3643e-05 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807681+0000", 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014696699999999999 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807738+0000", 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6885999999999997e-05 2024-09-16T04:08:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.843000931999999, 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67354323000000005, 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.453 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9500000000000002e-07 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.454 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509027+0000", 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4429999999999999e-06 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181881+0000", 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67285377899999999 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181891+0000", 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0479000000000001e-05 2024-09-16T04:08:49.455 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181961+0000", 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0370000000000006e-05 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182566+0000", 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060456400000000003 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.456 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.4s1 62/56 ECSubRead(tid=8, to_read={3:26eb5942:::smithi01639990-23:head=0,4096,0}, subchunks={3:26eb5942:::smithi01639990-23:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.704370102, 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64312231099999995, 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.457 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2500000000000004e-07 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.458 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647678+0000", 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3421000000000002e-05 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290437+0000", 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64275884900000002 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290450+0000", 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2858e-05 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.459 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290545+0000", 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5463000000000004e-05 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290776+0000", 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000230895 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:705 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:49.460 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.85928365299999998, 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.037755205, 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 705 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.461 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492741+0000", 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6199999999999997e-07 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.462 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492743+0000", 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7530000000000002e-06 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492750+0000", 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7710000000000001e-06 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492756+0000", 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.524e-06 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.463 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494038+0000", 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012823310000000001 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494140+0000", 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010216400000000001 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494664+0000", 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052402199999999996 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.464 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495429+0000", 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00076470399999999995 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530435+0000", 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035005738000000002 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530495+0000", 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0535999999999998e-05 2024-09-16T04:08:49.465 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:706 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.85925792899999998, 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.039505024, 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:49.466 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 706 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.467 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0600000000000001e-07 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492769+0000", 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7420000000000002e-06 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492776+0000", 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0480000000000001e-06 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.468 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492781+0000", 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1499999999999998e-06 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495503+0000", 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027220199999999999 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495628+0000", 2024-09-16T04:08:49.469 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000125041 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530799+0000", 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035171257999999997 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530897+0000", 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7608999999999993e-05 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:49.470 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.532271+0000", 2024-09-16T04:08:49.471 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00137375 2024-09-16T04:08:49.471 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.471 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.471 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.471 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:49.471 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:49.471 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:49.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:08:49.473 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:49.473 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:49.473 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-16T04:08:49.474 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:49.474 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:49.470+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:49.474 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:49.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:49.475 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:49.475 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:49.476 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:49.476 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:49.476 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:49.477 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:49.477 INFO:tasks.rados.rados.0.smithi016.stdout:295: done (3 left) 2024-09-16T04:08:49.477 INFO:tasks.rados.rados.0.smithi016.stdout:296: snap_create 2024-09-16T04:08:49.477 INFO:tasks.rados.rados.0.smithi016.stdout:294: finishing copy_from racing read to smithi01639990-1 2024-09-16T04:08:49.477 INFO:tasks.rados.rados.0.smithi016.stdout:292: finishing write tid 5 to smithi01639990-47 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:292: finishing write tid 6 to smithi01639990-47 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:292: oid 47 updating version 0 to 58 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:292: oid 47 version 58 is already newer than 56 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:292: oid 47 updating version 58 to 59 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:292: oid 47 version 59 is already newer than 57 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:292: oid 47 updating version 59 to 60 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 47 v 60 (ObjNum 128 snap 23 seq_num 128) dirty exists 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:292: left oid 47 (ObjNum 128 snap 23 seq_num 128) 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:294: finishing copy_from to smithi01639990-1 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 1 v 69 (ObjNum 117 snap 20 seq_num 117) dirty exists 2024-09-16T04:08:49.478 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 26 v 25 (ObjNum 80 snap 7 seq_num 80) dirty exists 2024-09-16T04:08:49.480 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:49.480 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:49.505 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:49.505 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:49.531 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:49.552 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:49.573 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:49.571+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:49.574 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:49.574 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:49.590 INFO:teuthology.orchestra.run.smithi016.stderr:no valid command found; 10 closest matches: 2024-09-16T04:08:49.590 INFO:teuthology.orchestra.run.smithi016.stderr:0 2024-09-16T04:08:49.590 INFO:teuthology.orchestra.run.smithi016.stderr:1 2024-09-16T04:08:49.590 INFO:teuthology.orchestra.run.smithi016.stderr:2 2024-09-16T04:08:49.590 INFO:teuthology.orchestra.run.smithi016.stderr:abort 2024-09-16T04:08:49.590 INFO:teuthology.orchestra.run.smithi016.stderr:assert 2024-09-16T04:08:49.590 INFO:teuthology.orchestra.run.smithi016.stderr:bluestore allocator dump block 2024-09-16T04:08:49.590 INFO:teuthology.orchestra.run.smithi016.stderr:bluestore allocator fragmentation block 2024-09-16T04:08:49.591 INFO:teuthology.orchestra.run.smithi016.stderr:bluestore allocator fragmentation histogram block [] [] 2024-09-16T04:08:49.591 INFO:teuthology.orchestra.run.smithi016.stderr:bluestore allocator score block 2024-09-16T04:08:49.591 INFO:teuthology.orchestra.run.smithi016.stderr:config diff 2024-09-16T04:08:49.591 INFO:teuthology.orchestra.run.smithi016.stderr:admin_socket: invalid command 2024-09-16T04:08:49.591 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:49.591 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T04:08:49.593 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-16T04:08:49.596 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T04:08:49.601 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T04:08:49.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:49.674 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:49.672+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:49.748 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:49.748 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:49.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:49.774 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:49.773+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:49.827 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:49.825+0000 7fea06ae8640 -1 osd.1 54 log_to_monitors true 2024-09-16T04:08:49.873 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:49.873 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:49.875 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:49.874+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:49.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:49.976 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:49.975+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:49.990 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:49.990 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:49.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:50.077 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:50.075+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:50.107 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:50.107 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:50.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:50.178 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:50.177+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:50.222 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:50.223 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:50.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:50.278 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:50.277+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:50.338 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:50.338 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:50.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:50.379 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:50.378+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:50.437 INFO:tasks.rados.rados.0.smithi016.stdout:292: done (3 left) 2024-09-16T04:08:50.437 INFO:tasks.rados.rados.0.smithi016.stdout:293: done (2 left) 2024-09-16T04:08:50.437 INFO:tasks.rados.rados.0.smithi016.stdout:294: done (1 left) 2024-09-16T04:08:50.437 INFO:tasks.rados.rados.0.smithi016.stdout:296: done (0 left) 2024-09-16T04:08:50.437 INFO:tasks.rados.rados.0.smithi016.stdout:297: read oid 3 snap 23 2024-09-16T04:08:50.437 INFO:tasks.rados.rados.0.smithi016.stdout:297: expect (ObjNum 126 snap 22 seq_num 126) 2024-09-16T04:08:50.437 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 49 current snap is 24 2024-09-16T04:08:50.438 INFO:tasks.rados.rados.0.smithi016.stdout:298: seq_num 129 ranges {2310144=622592,2932736=655360,3588096=688128,4276224=212992} 2024-09-16T04:08:50.449 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:50.447+0000 7fe9fe89f640 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:08:50.453 INFO:tasks.rados.rados.0.smithi016.stdout:298: writing smithi01639990-49 from 2310144 to 2932736 tid 1 2024-09-16T04:08:50.457 INFO:tasks.rados.rados.0.smithi016.stdout:298: writing smithi01639990-49 from 2932736 to 3588096 tid 2 2024-09-16T04:08:50.462 INFO:tasks.rados.rados.0.smithi016.stdout:298: writing smithi01639990-49 from 3588096 to 4276224 tid 3 2024-09-16T04:08:50.464 INFO:tasks.rados.rados.0.smithi016.stdout:298: writing smithi01639990-49 from 4276224 to 4489216 tid 4 2024-09-16T04:08:50.464 INFO:tasks.rados.rados.0.smithi016.stdout:299: rollback oid 40 current snap is 24 2024-09-16T04:08:50.464 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 40 to 19 2024-09-16T04:08:50.464 INFO:tasks.rados.rados.0.smithi016.stdout:298: finishing write tid 1 to smithi01639990-49 2024-09-16T04:08:50.464 INFO:tasks.rados.rados.0.smithi016.stdout:300: read oid 4 snap -1 2024-09-16T04:08:50.464 INFO:tasks.rados.rados.0.smithi016.stdout:300: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:08:50.464 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 32 current snap is 24 2024-09-16T04:08:50.465 INFO:tasks.rados.rados.0.smithi016.stdout:301: seq_num 130 ranges {0=638976,638976=671744,1310720=802816,2113536=81920} 2024-09-16T04:08:50.468 INFO:tasks.rados.rados.0.smithi016.stdout:301: writing smithi01639990-32 from 0 to 638976 tid 1 2024-09-16T04:08:50.472 INFO:tasks.rados.rados.0.smithi016.stdout:301: writing smithi01639990-32 from 638976 to 1310720 tid 2 2024-09-16T04:08:50.477 INFO:tasks.rados.rados.0.smithi016.stdout:301: writing smithi01639990-32 from 1310720 to 2113536 tid 3 2024-09-16T04:08:50.477 INFO:tasks.rados.rados.0.smithi016.stdout:301: writing smithi01639990-32 from 2113536 to 2195456 tid 4 2024-09-16T04:08:50.478 INFO:tasks.rados.rados.0.smithi016.stdout:302: snap_create 2024-09-16T04:08:50.478 INFO:tasks.rados.rados.0.smithi016.stdout:300: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T04:08:50.480 INFO:tasks.rados.rados.0.smithi016.stdout:298: finishing write tid 2 to smithi01639990-49 2024-09-16T04:08:50.480 INFO:tasks.rados.rados.0.smithi016.stdout:298: finishing write tid 3 to smithi01639990-49 2024-09-16T04:08:50.480 INFO:tasks.rados.rados.0.smithi016.stdout:299: finishing rollback tid 1 to smithi01639990-40 2024-09-16T04:08:50.480 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 40 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-16T04:08:50.480 INFO:tasks.rados.rados.0.smithi016.stdout:298: finishing write tid 4 to smithi01639990-49 2024-09-16T04:08:50.480 INFO:tasks.rados.rados.0.smithi016.stdout:298: finishing write tid 5 to smithi01639990-49 2024-09-16T04:08:50.480 INFO:tasks.rados.rados.0.smithi016.stdout:301: finishing write tid 1 to smithi01639990-32 2024-09-16T04:08:50.480 INFO:tasks.rados.rados.0.smithi016.stdout:298: finishing write tid 6 to smithi01639990-49 2024-09-16T04:08:50.480 INFO:tasks.rados.rados.0.smithi016.stdout:298: oid 49 updating version 0 to 93 2024-09-16T04:08:50.481 INFO:tasks.rados.rados.0.smithi016.stdout:298: oid 49 version 93 is already newer than 92 2024-09-16T04:08:50.481 INFO:tasks.rados.rados.0.smithi016.stdout:298: oid 49 version 93 is already newer than 91 2024-09-16T04:08:50.481 INFO:tasks.rados.rados.0.smithi016.stdout:298: oid 49 version 93 is already newer than 89 2024-09-16T04:08:50.481 INFO:tasks.rados.rados.0.smithi016.stdout:298: oid 49 version 93 is already newer than 90 2024-09-16T04:08:50.481 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 49 v 93 (ObjNum 129 snap 24 seq_num 129) dirty exists 2024-09-16T04:08:50.481 INFO:tasks.rados.rados.0.smithi016.stdout:298: left oid 49 (ObjNum 129 snap 24 seq_num 129) 2024-09-16T04:08:50.481 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:50.478+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:50.486 INFO:tasks.rados.rados.0.smithi016.stdout:301: finishing write tid 2 to smithi01639990-32 2024-09-16T04:08:50.486 INFO:tasks.rados.rados.0.smithi016.stdout:301: finishing write tid 3 to smithi01639990-32 2024-09-16T04:08:50.486 INFO:tasks.rados.rados.0.smithi016.stdout:301: finishing write tid 4 to smithi01639990-32 2024-09-16T04:08:50.486 INFO:tasks.rados.rados.0.smithi016.stdout:301: finishing write tid 5 to smithi01639990-32 2024-09-16T04:08:50.488 INFO:tasks.rados.rados.0.smithi016.stdout:301: finishing write tid 6 to smithi01639990-32 2024-09-16T04:08:50.488 INFO:tasks.rados.rados.0.smithi016.stdout:301: oid 32 updating version 0 to 74 2024-09-16T04:08:50.488 INFO:tasks.rados.rados.0.smithi016.stdout:301: oid 32 version 74 is already newer than 71 2024-09-16T04:08:50.488 INFO:tasks.rados.rados.0.smithi016.stdout:301: oid 32 version 74 is already newer than 70 2024-09-16T04:08:50.489 INFO:tasks.rados.rados.0.smithi016.stdout:301: oid 32 version 74 is already newer than 73 2024-09-16T04:08:50.489 INFO:tasks.rados.rados.0.smithi016.stdout:301: oid 32 version 74 is already newer than 72 2024-09-16T04:08:50.489 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 32 v 74 (ObjNum 130 snap 24 seq_num 130) dirty exists 2024-09-16T04:08:50.489 INFO:tasks.rados.rados.0.smithi016.stdout:301: left oid 32 (ObjNum 130 snap 24 seq_num 130) 2024-09-16T04:08:50.510 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:50.510 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:50.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:50.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:08:50.581 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:50.580+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:50.667 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:50.668 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:50.678 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:50.682 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:50.681+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:50.728 INFO:tasks.rados.rados.0.smithi016.stdout:297: expect (ObjNum 126 snap 22 seq_num 126) 2024-09-16T04:08:50.785 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:50.782+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:50.789 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:50.789 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:50.798 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:50.884 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:50.883+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:50.943 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:50.943 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:50.952 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:50.996 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:50.995+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:51.056 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:51.056 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:51.065 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:51.107 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:51.105+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:51.187 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:51.185+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:51.210 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:51.210 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:51.219 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:51.287 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:51.286+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:51.321 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:51.321 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:51.330 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:51.389 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:51.388+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:51.433 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:51.433 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:51.435 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-16T04:08:51.438 INFO:tasks.rados.rados.0.smithi016.stdout:297: done (5 left) 2024-09-16T04:08:51.438 INFO:tasks.rados.rados.0.smithi016.stdout:298: done (4 left) 2024-09-16T04:08:51.438 INFO:tasks.rados.rados.0.smithi016.stdout:299: done (3 left) 2024-09-16T04:08:51.439 INFO:tasks.rados.rados.0.smithi016.stdout:300: done (2 left) 2024-09-16T04:08:51.439 INFO:tasks.rados.rados.0.smithi016.stdout:301: done (1 left) 2024-09-16T04:08:51.439 INFO:tasks.rados.rados.0.smithi016.stdout:302: done (0 left) 2024-09-16T04:08:51.439 INFO:tasks.rados.rados.0.smithi016.stdout:303: snap_remove snap 23 2024-09-16T04:08:51.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:51.489 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:51.487+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:51.588 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:51.588 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:51.590 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:51.588+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:51.596 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:51.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:51.691 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:51.690+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:51.738 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:51.739 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:51.740 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:51.740 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:51.749 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:51.759 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:51.760 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:51.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:51.781 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:51.793 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:51.792+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:51.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:51.834 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:51.853 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:51.853 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:51.854 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:51.855 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:51.862 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:51.876 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:51.898 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:51.898 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:51.905 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:51.903+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:51.920 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:51.920 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:51.942 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:51.942 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:51.963 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:51.964 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:51.985 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:51.985 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:51.995 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:51.993+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:52.007 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:52.007 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:52.007 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:52.007 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:52.016 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:52.095 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:52.094+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:52.119 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:52.119 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:52.119 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:52.119 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:52.119 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:52.128 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:52.196 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:52.194+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:52.233 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 62.289991868000001, 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:52.234 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:52.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:52.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:52.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:52.241 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.307587624999996, 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:52.242 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:52.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:52.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:52.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:52.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.297225118999997, 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:52.248 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:52.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:52.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 59.800637066999997, 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:52.252 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:52.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:52.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:52.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.257 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 59.346062574000001, 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:52.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:52.259 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:52.263 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 59.346008513000001, 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:52.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.121739974999997, 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.269 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.272 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.543806015000001, 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:52.273 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.274 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.275 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.276 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s2 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.131031786000001, 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010866819, 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.277 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099683+0000", 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099684+0000", 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.004e-06 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099714+0000", 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9807000000000001e-05 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099769+0000", 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5306000000000002e-05 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.279 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099781+0000", 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2091e-05 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099835+0000", 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4058000000000003e-05 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099845+0000", 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5729999999999997e-06 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.280 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110509+0000", 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010664339 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110550+0000", 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0640999999999999e-05 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.281 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.126992870000002, 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.282 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.283 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:52.284 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.285 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 43.125605456000002, 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.286 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:52.287 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:52.288 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.289 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=188, reqid=client.4440.0:379, at_version=47'56, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:52.290 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.778473998999999, 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011758437, 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:52.291 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452266+0000", 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4151e-05 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.292 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459768+0000", 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075025179999999997 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459778+0000", 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0244999999999999e-05 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459835+0000", 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6394999999999998e-05 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.293 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459845+0000", 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0518e-05 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463963+0000", 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041182450000000004 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463999+0000", 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5667999999999998e-05 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.294 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.766286426000001, 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.295 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.296 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.297 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.298 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.793416476000001, 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019153804, 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.299 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.300 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437299+0000", 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437320+0000", 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1121999999999999e-05 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437388+0000", 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7889e-05 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.301 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437397+0000", 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4390000000000006e-06 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441658+0000", 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042607110000000004 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441667+0000", 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.302 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456416+0000", 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014748493 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456452+0000", 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6159999999999999e-05 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:52.303 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.790226765, 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017174697999999999, 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.304 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440489+0000", 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.015e-06 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440509+0000", 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0338000000000001e-05 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.305 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440547+0000", 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440557+0000", 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7820000000000005e-06 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442068+0000", 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015112179999999999 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.306 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442078+0000", 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4620000000000002e-06 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457639+0000", 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015560954 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457663+0000", 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4121000000000002e-05 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.307 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.as7 68/56 ECSubWrite(tid=57, reqid=osd.6.0:57, at_version=68'42, trim_to=0'0, roll_forward_to=64'40))", 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.6965595130000004, 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017528866000000001, 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.308 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.49e-07 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.309 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534179+0000", 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3308000000000001e-05 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546651+0000", 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012471873 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546660+0000", 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2229999999999996e-06 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.310 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546730+0000", 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0123000000000006e-05 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546741+0000", 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0684e-05 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551646+0000", 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049059180000000004 2024-09-16T04:08:52.311 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551684+0000", 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7588e-05 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 68/56 ECSubWriteReply(tid=200, last_complete=68'24, committed=1, applied=1))", 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.6675474890000004, 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46574167999999999, 2024-09-16T04:08:52.312 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.313 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563168+0000", 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4099999999999997e-07 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563174+0000", 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.327e-06 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028683+0000", 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46550948399999997 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.314 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028698+0000", 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4576000000000001e-05 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028858+0000", 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016062100000000001 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028909+0000", 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0730999999999999e-05 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.315 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:654 3.ds0 3:bc502819:::smithi01639990-5:head [create,setxattr XXWdXV_kiPaajZX\\PgmqNaZqSVmab^hrtRYeagZr (1894) in=40b,setxattr NdZr`e\\^kXiksh]\\XocenQrhYc]c`[proOfo`Pd^ (33) in=40b,setxattr _`NqSdldlQbUNl^ZNRba\\naZhhdraiOgpafrZb[o (103) in=40b,setxattr iadimrq_[jOPUblhnVqTYqQNedg (1647) in=27b,setxattr `e\\rtQi]nmoP\\cm]oRYN`[]^qVUTsU^_oXm]h\\ec (6452) in=40b,setxattr ppZXc_dYi_WWXcUdeqcpck_ceUVa^XftgnfgdP[q (3576) in=40b,setxattr hehZsP\\ajqhe^nfrZjQmsnfWoqq\\e\\NV^rOaZYtj (1787) in=40b,setxattr a]Smbg`^k`emWabQRYcpr_lQj`eh`[r\\riqa_sfd (1015) in=40b,setxattr eqsaZkZ[`t^R^grhYOVrVsspVQQp_Zo\\loUV[\\U[ (982) in=40b,setxattr Rn`k`cSYQj`XOZdn_mSPrqQtoUpWTk[^[t (1647) in=34b,setxattr ]TtVabQgUTabfmqae[UVe`cNsqqYfdh]\\sVoeXQr (383) in=40b,setxattr iRn][N_NY\\ZhmYdQ[[ptTP_NW^Z]_roZnOpcmR_Y (19016) in=40b,setxattr T_b_WhnZnkaaYkNmmOUnaZa`dhRhQSshYQVQnVql (13472) in=40b,setxattr \\hZXh`dT^W[c\\XodZZ_d\\QO\\\\[PVWthO_WVOPSQd (1681) in=40b,setxattr ctSOqNY_dUqTOjgXYjqgslgNRRPoT\\\\OXPPWrtt_ (522) in=40b,setxattr f\\hTOf^sbRaYeUntSV^WhUo]qV`bSNermptfXs]Q (1645) in=40b,setxattr ohNTdfibmYs`eUTiQa^bcYW\\aarhT]YQ_haagqpf (1119) in=40b,setxattr qqUtZZ]ataUj_[pnh\\jVtU_dRpi^g[VPNeS`Y\\hk (699) in=40b,setxattr qsgedRfqdW]]cYbWOUgh^ZleaSQiteejr^afXbUj (525) in=40b,setxattr [XmPqpoSg`fWjdlmQ_qabof]_WVcPaY[]k[ardgj (594) in=40b] snapc 15=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 7.6965364540000003, 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019213674, 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 654 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534181+0000", 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5830000000000001e-06 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534398+0000", 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000216663 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534421+0000", 2024-09-16T04:08:52.318 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3714e-05 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534434+0000", 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3185e-05 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534504+0000", 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9665000000000005e-05 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534510+0000", 2024-09-16T04:08:52.319 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1709999999999999e-06 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.538902+0000", 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043914660000000001 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539044+0000", 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014215000000000001 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.320 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540400+0000", 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013559189999999999 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541692+0000", 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001291964 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553277+0000", 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011585432999999999 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.321 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553392+0000", 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011476099999999999 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.bs5 69/56 ECSubWrite(tid=338, reqid=osd.3.0:338, at_version=69'72, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.544515+0000", 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 7.6861997750000004, 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011654167, 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.322 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544515+0000", 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544515+0000", 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544516+0000", 2024-09-16T04:08:52.323 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544539+0000", 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2705e-05 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544591+0000", 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2037999999999999e-05 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544609+0000", 2024-09-16T04:08:52.324 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546316+0000", 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001707456 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546327+0000", 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1025e-05 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.325 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.556143+0000", 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098162240000000001 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.556169+0000", 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5633e-05 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 74/48 ECSubWrite(tid=134, reqid=osd.8.0:134, at_version=74'63, trim_to=0'0, roll_forward_to=73'61))", 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:49.453276+0000", 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 2.7774384759999999, 2024-09-16T04:08:52.326 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011031914, 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453276+0000", 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453276+0000", 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.327 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453277+0000", 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6100000000000002e-07 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453304+0000", 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7410999999999999e-05 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453370+0000", 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6082999999999997e-05 2024-09-16T04:08:52.328 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453379+0000", 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8869999999999998e-06 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453447+0000", 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8403000000000002e-05 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453458+0000", 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0715e-05 2024-09-16T04:08:52.329 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.463659+0000", 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010200799 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.464308+0000", 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064935500000000001 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:52.330 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:52.331 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:52.332 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:52.333 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:52.330+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:52.433 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:52.431+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:52.439 INFO:tasks.rados.rados.0.smithi016.stdout:303: done (0 left) 2024-09-16T04:08:52.439 INFO:tasks.rados.rados.0.smithi016.stdout:304: rollback oid 10 current snap is 25 2024-09-16T04:08:52.440 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 10 to 24 2024-09-16T04:08:52.440 INFO:tasks.rados.rados.0.smithi016.stdout:305: copy_from oid 38 from oid 1 current snap is 25 2024-09-16T04:08:52.440 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 46 current snap is 25 2024-09-16T04:08:52.440 INFO:tasks.rados.rados.0.smithi016.stdout:306: seq_num 131 ranges {2162688=655360,2818048=425984,3244032=638976,3883008=491520} 2024-09-16T04:08:52.455 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:52.455 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:52.455 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.455 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(osd.6.19:3 3.15s0 3:ada5d29c:::smithi01639990-1:head [assert-version v69,copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:52.449012+0000", 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.0050385940000000004, 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0050685440000000003, 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:52.449012+0000", 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.456 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:52.449012+0000", 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:52.449014+0000", 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.689e-06 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:52.449019+0000", 2024-09-16T04:08:52.457 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7859999999999996e-06 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:52.449401+0000", 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00038194500000000002 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:52.449407+0000", 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5100000000000004e-06 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.458 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:52.449468+0000", 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0421000000000003e-05 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:52.449573+0000", 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105287 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 1 2024-09-16T04:08:52.459 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:52.460 INFO:tasks.rados.rados.0.smithi016.stdout:306: writing smithi01639990-46 from 2162688 to 2818048 tid 1 2024-09-16T04:08:52.460 INFO:tasks.rados.rados.0.smithi016.stdout:306: writing smithi01639990-46 from 2818048 to 3244032 tid 2 2024-09-16T04:08:52.463 INFO:tasks.rados.rados.0.smithi016.stdout:306: writing smithi01639990-46 from 3244032 to 3883008 tid 3 2024-09-16T04:08:52.467 INFO:tasks.rados.rados.0.smithi016.stdout:306: writing smithi01639990-46 from 3883008 to 4374528 tid 4 2024-09-16T04:08:52.467 INFO:tasks.rados.rados.0.smithi016.stdout:307: snap_create 2024-09-16T04:08:52.467 INFO:tasks.rados.rados.0.smithi016.stdout:304: finishing rollback tid 0 to smithi01639990-10 2024-09-16T04:08:52.468 INFO:tasks.rados.rados.0.smithi016.stdout:304: finishing rollback tid 1 to smithi01639990-10 2024-09-16T04:08:52.468 INFO:tasks.rados.rados.0.smithi016.stdout:304: finishing rollback tid 2 to smithi01639990-10 2024-09-16T04:08:52.468 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 10 v 12 (ObjNum 87 snap 11 seq_num 87) dirty exists 2024-09-16T04:08:52.468 INFO:tasks.rados.rados.0.smithi016.stdout:305: finishing copy_from racing read to smithi01639990-38 2024-09-16T04:08:52.468 INFO:tasks.rados.rados.0.smithi016.stdout:306: finishing write tid 1 to smithi01639990-46 2024-09-16T04:08:52.468 INFO:tasks.rados.rados.0.smithi016.stdout:306: finishing write tid 2 to smithi01639990-46 2024-09-16T04:08:52.470 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:52.471 INFO:tasks.rados.rados.0.smithi016.stdout:305: finishing copy_from to smithi01639990-38 2024-09-16T04:08:52.471 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 38 v 69 (ObjNum 117 snap 20 seq_num 117) dirty exists 2024-09-16T04:08:52.472 INFO:tasks.rados.rados.0.smithi016.stdout:306: finishing write tid 3 to smithi01639990-46 2024-09-16T04:08:52.472 INFO:tasks.rados.rados.0.smithi016.stdout:306: finishing write tid 4 to smithi01639990-46 2024-09-16T04:08:52.493 INFO:tasks.rados.rados.0.smithi016.stdout:306: finishing write tid 5 to smithi01639990-46 2024-09-16T04:08:52.495 INFO:tasks.rados.rados.0.smithi016.stdout:306: finishing write tid 6 to smithi01639990-46 2024-09-16T04:08:52.495 INFO:tasks.rados.rados.0.smithi016.stdout:306: oid 46 updating version 0 to 93 2024-09-16T04:08:52.495 INFO:tasks.rados.rados.0.smithi016.stdout:306: oid 46 version 93 is already newer than 92 2024-09-16T04:08:52.495 INFO:tasks.rados.rados.0.smithi016.stdout:306: oid 46 updating version 93 to 95 2024-09-16T04:08:52.495 INFO:tasks.rados.rados.0.smithi016.stdout:306: oid 46 updating version 95 to 96 2024-09-16T04:08:52.495 INFO:tasks.rados.rados.0.smithi016.stdout:306: oid 46 version 96 is already newer than 94 2024-09-16T04:08:52.495 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 46 v 96 (ObjNum 131 snap 25 seq_num 131) dirty exists 2024-09-16T04:08:52.495 INFO:tasks.rados.rados.0.smithi016.stdout:306: left oid 46 (ObjNum 131 snap 25 seq_num 131) 2024-09-16T04:08:52.533 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:52.532+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:52.574 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:52.575 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:52.575 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:52.583 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:52.635 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:52.633+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:52.729 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:52.729 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:52.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:52.729 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:52.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.729 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:52.729 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:52.729 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 43.584550526999998, 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:52.733 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.266494919000003, 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:08:52.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.135952760000002, 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:08:52.742 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.055030207000001, 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:52.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.894626149, 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.747 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:08:52.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.218521312, 2024-09-16T04:08:52.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:08:52.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.756 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.196808287, 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:52.757 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:08:52.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:52.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:52.762 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.196756059999998, 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:52.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.767 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 67/56 ECSubWrite(tid=320, reqid=osd.3.0:320, at_version=67'62, trim_to=0'0, roll_forward_to=66'60))", 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.193695086, 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01263539, 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6899999999999999e-07 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532569+0000", 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0305999999999998e-05 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532682+0000", 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112361 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532692+0000", 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0832000000000001e-05 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532753+0000", 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0568e-05 2024-09-16T04:08:52.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532763+0000", 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6010000000000001e-06 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545152+0000", 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012389908 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545184+0000", 2024-09-16T04:08:52.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1644999999999999e-05 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.11s6 68/56 ECSubWrite(tid=17, reqid=osd.10.0:17, at_version=68'19, trim_to=0'0, roll_forward_to=64'17))", 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.1870701379999993, 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014242092, 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.772 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8500000000000001e-07 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539199+0000", 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4397999999999999e-05 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539248+0000", 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9033000000000003e-05 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539258+0000", 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8800000000000003e-06 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539858+0000", 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060046199999999996 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539867+0000", 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.363e-06 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553388+0000", 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013520249 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553416+0000", 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8121999999999999e-05 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.776 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 68/56 ECSubWrite(tid=327, reqid=osd.3.0:327, at_version=68'66, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.1867786799999998, 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012797307000000001, 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539466+0000", 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4000000000000001e-07 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539484+0000", 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8391999999999999e-05 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539516+0000", 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1439999999999997e-05 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539524+0000", 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4570000000000005e-06 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542244+0000", 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027199070000000001 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542254+0000", 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8500000000000006e-06 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552231+0000", 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0099774170000000006 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552262+0000", 2024-09-16T04:08:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1204000000000001e-05 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=64, reqid=osd.4.0:64, at_version=68'43, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.1859561339999996, 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014089379000000001, 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1500000000000001e-07 2024-09-16T04:08:52.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540308+0000", 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0585999999999999e-05 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540337+0000", 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8560000000000001e-05 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540345+0000", 2024-09-16T04:08:52.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6299999999999998e-06 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540400+0000", 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4818000000000003e-05 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540411+0000", 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1009000000000001e-05 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554349+0000", 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013938394 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554377+0000", 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8167e-05 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=66, reqid=osd.4.0:66, at_version=68'45, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.1841148429999997, 2024-09-16T04:08:52.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01391278, 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:08:52.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542130+0000", 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6600000000000005e-07 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542154+0000", 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3862999999999998e-05 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542183+0000", 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9108999999999999e-05 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542192+0000", 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549962+0000", 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007769977 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549972+0000", 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0074e-05 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556017+0000", 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060444319999999998 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556042+0000", 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5012999999999999e-05 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.789 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s7 68/56 ECSubWrite(tid=197, reqid=osd.7.0:197, at_version=68'24, trim_to=0'0, roll_forward_to=64'22))", 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.1837714080000001, 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013604978, 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542473+0000", 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5099999999999999e-07 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542491+0000", 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8417e-05 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542561+0000", 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9744000000000002e-05 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542568+0000", 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.289e-06 2024-09-16T04:08:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546547+0000", 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039792350000000002 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546557+0000", 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8870000000000004e-06 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556052+0000", 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094947169999999997 2024-09-16T04:08:52.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556077+0000", 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5537999999999998e-05 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=119, reqid=client.4440.0:705, at_version=73'60, trim_to=0'0, roll_forward_to=73'57))", 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.2315587680000002, 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035584957, 2024-09-16T04:08:52.794 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494686+0000", 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9999999999999996e-07 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494705+0000", 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9208999999999999e-05 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529770+0000", 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035064924999999997 2024-09-16T04:08:52.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529777+0000", 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5069999999999997e-06 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529831+0000", 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4122000000000001e-05 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529845+0000", 2024-09-16T04:08:52.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4571e-05 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530243+0000", 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039740100000000003 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530270+0000", 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7421999999999999e-05 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:708 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 69] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.2144892279999997, 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49730772000000001, 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 708 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.799 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511756+0000", 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6029999999999999e-06 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511762+0000", 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2500000000000003e-06 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511774+0000", 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1761e-05 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511797+0000", 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2416999999999999e-05 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511872+0000", 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5437000000000003e-05 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511975+0000", 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010341899999999999 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.526578+0000", 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014602084 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529422+0000", 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028444880000000001 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008984+0000", 2024-09-16T04:08:52.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47956190999999998 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009062+0000", 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8350999999999996e-05 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:709 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:08:52.804 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.2144337260000002, 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017140263999999999, 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 709 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.805 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0800000000000001e-07 2024-09-16T04:08:52.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511812+0000", 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7799999999999999e-06 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511816+0000", 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.172e-06 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511819+0000", 2024-09-16T04:08:52.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9739999999999998e-06 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512187+0000", 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036796799999999998 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512240+0000", 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3013999999999998e-05 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528792+0000", 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016551461999999999 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528863+0000", 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0797000000000007e-05 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528950+0000", 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7788999999999996e-05 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.809 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.810 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.810 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.810 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:08:52.810 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:08:52.810 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.19561721, 2024-09-16T04:08:52.810 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.36295003300000001, 2024-09-16T04:08:52.811 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.811 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.811 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.811 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:08:52.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7899999999999998e-07 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530637+0000", 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892881+0000", 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.362243749 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892890+0000", 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5620000000000004e-06 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892950+0000", 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0270999999999999e-05 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892959+0000", 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1689999999999998e-06 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:52.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893539+0000", 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000579628 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893577+0000", 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8056999999999998e-05 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=131, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:08:52.815 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.1953326730000002, 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47861527599999998, 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530912+0000", 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4400000000000001e-07 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530915+0000", 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5530000000000001e-06 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008849+0000", 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47793379000000002 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008854+0000", 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0340000000000004e-06 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008927+0000", 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3088000000000002e-05 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009526+0000", 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059936699999999996 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=132, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.7159134030000001, 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44531548500000001, 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:52.819 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010331+0000", 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9600000000000001e-07 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010332+0000", 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.454e-06 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455554+0000", 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44522236199999998 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455561+0000", 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4189999999999999e-06 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455608+0000", 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6928000000000002e-05 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455646+0000", 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8126000000000001e-05 2024-09-16T04:08:52.822 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.823 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.823 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.823 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:52.823 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:52.823 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:52.825 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:52.825 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:52.822+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:52.925 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:52.923+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:52.969 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:52.969 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:52.969 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:52.969 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:52.978 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:53.026 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:53.024+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:53.080 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:53.080 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:53.080 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:53.080 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:53.080 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:53.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:53.134 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:53.132+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 46.098088079999997, 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.195 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:53.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:53.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.762571110000003, 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.302620883000003, 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.203 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:08:53.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.979108277999998, 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.211 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.721324419999998, 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078488229999999996, 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471072+0000", 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0279999999999999e-06 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471418+0000", 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034541800000000001 2024-09-16T04:08:53.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471457+0000", 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471492+0000", 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471798+0000", 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030615599999999997 2024-09-16T04:08:53.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471812+0000", 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.448e-05 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478881+0000", 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070685549999999998 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478920+0000", 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9072000000000001e-05 2024-09-16T04:08:53.215 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.752209952000001, 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084259260000000002, 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.91e-07 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440211+0000", 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4978999999999999e-05 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440257+0000", 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6569000000000001e-05 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440268+0000", 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1056000000000001e-05 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440348+0000", 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9795000000000004e-05 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440364+0000", 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5996999999999999e-05 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448570+0000", 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082059590000000005 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448611+0000", 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.138e-05 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.bs7 57/56 [PushOp(3:d683cb60:::smithi01639990-49:head, version: 53'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d683cb60:::smithi01639990-49:head@53'38, size: 2310144, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:53.220 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.631627780999999, 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.043702234999999999, 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560769+0000", 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0049999999999999e-06 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.561301+0000", 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053276500000000004 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603955+0000", 2024-09-16T04:08:53.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042653605999999997 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603967+0000", 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2065000000000001e-05 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604063+0000", 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5599000000000004e-05 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604470+0000", 2024-09-16T04:08:53.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040719500000000001 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=252, reqid=osd.3.0:252, at_version=57'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.282901640999999, 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27497091299999998, 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.224 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909517+0000", 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2886000000000001e-05 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183847+0000", 2024-09-16T04:08:53.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27432981400000001 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183853+0000", 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2929999999999997e-06 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183902+0000", 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9008999999999998e-05 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183911+0000", 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184435+0000", 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052493699999999995 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184465+0000", 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9292e-05 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.228 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=253, reqid=osd.3.0:253, at_version=57'67, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.282273610000001, 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50851210899999999, 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910123+0000", 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910136+0000", 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.331e-05 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416910+0000", 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50677369000000005 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416919+0000", 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8290000000000003e-06 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416975+0000", 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5983999999999998e-05 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416985+0000", 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418599+0000", 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001614515 2024-09-16T04:08:53.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418634+0000", 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4588999999999997e-05 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 64/56 ECSubWrite(tid=285, reqid=client.4440.0:535, at_version=0'0, trim_to=0'0, roll_forward_to=64'72))", 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 13.650826541000001, 2024-09-16T04:08:53.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012095003, 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541570+0000", 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.173e-06 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541582+0000", 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2106999999999999e-05 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552877+0000", 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011294692 2024-09-16T04:08:53.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552889+0000", 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1875000000000001e-05 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552958+0000", 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9510000000000004e-05 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552967+0000", 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6240000000000008e-06 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553631+0000", 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066445299999999996 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553664+0000", 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2568999999999999e-05 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 68/46 ECSubWrite(tid=65, reqid=osd.4.0:65, at_version=68'22, trim_to=0'0, roll_forward_to=67'20))", 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.6511002829999999, 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010787494999999999, 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541313+0000", 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7431e-05 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541369+0000", 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6311999999999999e-05 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541376+0000", 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5729999999999996e-06 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544486+0000", 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031104150000000001 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544495+0000", 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0960000000000007e-06 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552010+0000", 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075148200000000002 2024-09-16T04:08:53.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552083+0000", 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2661000000000001e-05 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 68/56 ECSubWrite(tid=329, reqid=osd.3.0:329, at_version=68'68, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.6496465629999992, 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093665009999999993, 2024-09-16T04:08:53.242 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542750+0000", 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.23e-07 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542770+0000", 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0679999999999999e-05 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542842+0000", 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1391999999999997e-05 2024-09-16T04:08:53.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542850+0000", 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2320000000000001e-06 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545061+0000", 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022114220000000002 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545070+0000", 2024-09-16T04:08:53.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6780000000000006e-06 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552088+0000", 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070176939999999997 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552115+0000", 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7679999999999999e-05 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds2 69/56 ECSubWrite(tid=201, reqid=client.4440.0:654, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.651808699, 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090194439999999997, 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1400000000000001e-07 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540777+0000", 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000189886 2024-09-16T04:08:53.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540803+0000", 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6466e-05 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540811+0000", 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0730000000000005e-06 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540879+0000", 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8003999999999995e-05 2024-09-16T04:08:53.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540889+0000", 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8460000000000003e-06 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.549571+0000", 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086822059999999996 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.549606+0000", 2024-09-16T04:08:53.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4749000000000002e-05 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=336, reqid=osd.3.0:336, at_version=69'70, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.6487025190000004, 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077691619999999996, 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.05e-07 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543708+0000", 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4431e-05 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543897+0000", 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018918 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543906+0000", 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3349999999999996e-06 2024-09-16T04:08:53.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543954+0000", 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7793999999999997e-05 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543963+0000", 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4259999999999992e-06 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551434+0000", 2024-09-16T04:08:53.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0074706800000000004 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551462+0000", 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8110999999999999e-05 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=338, reqid=osd.3.0:338, at_version=69'72, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.6480083350000001, 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010064081000000001, 2024-09-16T04:08:53.255 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544388+0000", 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9800000000000003e-07 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544409+0000", 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0993e-05 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544442+0000", 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2937e-05 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544449+0000", 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.588e-06 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547347+0000", 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028976599999999998 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547357+0000", 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6479999999999998e-06 2024-09-16T04:08:53.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554419+0000", 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070620819999999999 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554451+0000", 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2475000000000003e-05 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s6 69/56 ECSubWrite(tid=202, reqid=osd.7.0:202, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.6466715399999998, 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010752746000000001, 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:08:53.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545725+0000", 2024-09-16T04:08:53.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4199999999999995e-07 2024-09-16T04:08:53.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545750+0000", 2024-09-16T04:08:53.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4915000000000001e-05 2024-09-16T04:08:53.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546880+0000", 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011307870000000001 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546891+0000", 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0825000000000001e-05 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551495+0000", 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004604131 2024-09-16T04:08:53.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551504+0000", 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6230000000000003e-06 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556450+0000", 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049460210000000001 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556477+0000", 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6801999999999999e-05 2024-09-16T04:08:53.266 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds2 69/56 ECSubWrite(tid=203, reqid=osd.7.0:203, at_version=69'28, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.6466425040000008, 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078627160000000005, 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.91e-07 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545769+0000", 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5562000000000001e-05 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546923+0000", 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001154457 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546936+0000", 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3183000000000001e-05 2024-09-16T04:08:53.269 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547734+0000", 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00079775500000000004 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547744+0000", 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5489999999999995e-06 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.553577+0000", 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058334809999999997 2024-09-16T04:08:53.270 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.553616+0000", 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8438e-05 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 69/56 ECSubWrite(tid=340, reqid=osd.3.0:340, at_version=69'98, trim_to=0'0, roll_forward_to=68'96))", 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.554920+0000", 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.6374755969999999, 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075442290000000004, 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.271 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554920+0000", 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554920+0000", 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.272 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554921+0000", 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9800000000000003e-07 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554941+0000", 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0653e-05 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554976+0000", 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4981000000000003e-05 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.273 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554987+0000", 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0336000000000001e-05 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.555043+0000", 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6787999999999999e-05 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.555054+0000", 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0205e-05 2024-09-16T04:08:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.562428+0000", 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0073745640000000001 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.562464+0000", 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6004000000000003e-05 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.15s7 69/48 ECSubWrite(tid=117, reqid=osd.8.0:117, at_version=69'59, trim_to=0'0, roll_forward_to=68'57))", 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.557667+0000", 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.6347289660000008, 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076099649999999998, 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557667+0000", 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.276 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557667+0000", 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557667+0000", 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3099999999999997e-07 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557688+0000", 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0990000000000001e-05 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.277 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557718+0000", 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9444000000000001e-05 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557726+0000", 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7290000000000006e-06 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557845+0000", 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011856 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.278 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.557856+0000", 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0832999999999999e-05 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.564509+0000", 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0066528170000000001 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.565276+0000", 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00076796099999999995 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 74/27 ECSubWrite(tid=161, reqid=osd.5.0:161, at_version=74'47, trim_to=0'0, roll_forward_to=73'45))", 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:49.456560+0000", 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 3.7358354399999998, 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076668730000000003, 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.280 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456560+0000", 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456560+0000", 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456560+0000", 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8799999999999999e-07 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.281 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456580+0000", 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0119000000000002e-05 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456649+0000", 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8806000000000005e-05 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456656+0000", 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9240000000000001e-06 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.282 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456720+0000", 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4368000000000004e-05 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456730+0000", 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5759999999999995e-06 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.464193+0000", 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0074625689999999996 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.283 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.464227+0000", 2024-09-16T04:08:53.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4323000000000003e-05 2024-09-16T04:08:53.284 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.284 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.284 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.284 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.284 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.284 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:53.285 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:53.286 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:53.283+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:53.386 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:53.384+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:53.430 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:53.430 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:53.430 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:53.430 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:53.439 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:53.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:08:53.466 INFO:tasks.rados.rados.0.smithi016.stdout:304: done (3 left) 2024-09-16T04:08:53.466 INFO:tasks.rados.rados.0.smithi016.stdout:305: done (2 left) 2024-09-16T04:08:53.466 INFO:tasks.rados.rados.0.smithi016.stdout:306: done (1 left) 2024-09-16T04:08:53.466 INFO:tasks.rados.rados.0.smithi016.stdout:307: done (0 left) 2024-09-16T04:08:53.466 INFO:tasks.rados.rados.0.smithi016.stdout:308: snap_create 2024-09-16T04:08:53.486 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:53.485+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:53.541 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:53.541 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:53.541 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:53.541 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:53.542 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:53.550 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:53.594 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:53.593+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:53.654 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:53.654 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:53.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:53.654 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:53.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 48/46 ECSubWrite(tid=190, reqid=osd.3.0:190, at_version=48'58, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.185679743000001, 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088361580000000002, 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465644+0000", 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465645+0000", 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4399999999999999e-07 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465670+0000", 2024-09-16T04:08:53.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4875000000000002e-05 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465791+0000", 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012066900000000001 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465800+0000", 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4539999999999996e-06 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465872+0000", 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2127999999999998e-05 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.465881+0000", 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3129999999999999e-06 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474449+0000", 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085683760000000008 2024-09-16T04:08:53.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474480+0000", 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1498999999999998e-05 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 52/46 ECSubWrite(tid=210, reqid=client.4440.0:434, at_version=52'63, trim_to=0'0, roll_forward_to=50'60))", 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.133557217, 2024-09-16T04:08:53.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83621405000000004, 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8200000000000001e-07 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517790+0000", 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2540000000000001e-05 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352789+0000", 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83499984500000002 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352804+0000", 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4569999999999999e-05 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352928+0000", 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012396799999999999 2024-09-16T04:08:53.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352943+0000", 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353943+0000", 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099977500000000006 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353981+0000", 2024-09-16T04:08:53.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7700999999999997e-05 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=217, reqid=client.4440.0:449, at_version=53'58, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.140047181, 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0096255799999999999, 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.664 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511278+0000", 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.068e-06 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511623+0000", 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034493499999999999 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515041+0000", 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034178889999999999 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515048+0000", 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3479999999999998e-06 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515115+0000", 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6469000000000003e-05 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515126+0000", 2024-09-16T04:08:53.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1187e-05 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520872+0000", 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057460280000000002 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520902+0000", 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0655999999999999e-05 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.668 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.669 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.669 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=218, reqid=client.4440.0:450, at_version=53'59, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:53.669 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:53.672 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.136702606, 2024-09-16T04:08:53.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089217440000000005, 2024-09-16T04:08:53.672 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.672 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.672 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.672 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514621+0000", 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.514623+0000", 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2300000000000001e-06 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515021+0000", 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039790399999999998 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519384+0000", 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043635380000000001 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519391+0000", 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9709999999999998e-06 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519441+0000", 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9493999999999997e-05 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519449+0000", 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2130000000000008e-06 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.523504+0000", 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040551220000000004 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.523543+0000", 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9271999999999999e-05 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=220, reqid=client.4440.0:452, at_version=53'61, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:53.676 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.131553663999998, 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51827822899999998, 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519771+0000", 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.61e-07 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519788+0000", 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6699999999999999e-05 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036882+0000", 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51709414300000001 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036895+0000", 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3363999999999999e-05 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036964+0000", 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8734999999999998e-05 2024-09-16T04:08:53.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036973+0000", 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0289999999999994e-06 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038014+0000", 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010410529999999999 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038049+0000", 2024-09-16T04:08:53.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4443999999999998e-05 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:526 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~622592] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.020757276000001, 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01502261, 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:53.681 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 526 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:53.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630568+0000", 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5680000000000001e-06 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631353+0000", 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078490700000000005 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631367+0000", 2024-09-16T04:08:53.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3441000000000001e-05 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631375+0000", 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8809999999999996e-06 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631465+0000", 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0673000000000004e-05 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631565+0000", 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9753000000000005e-05 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.632641+0000", 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001076097 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634418+0000", 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017769680000000001 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645537+0000", 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011119393999999999 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645589+0000", 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1928000000000002e-05 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.686 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:527 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.019920203000002, 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015053211, 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 527 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631405+0000", 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.08e-06 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631411+0000", 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9009999999999999e-06 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631419+0000", 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631426+0000", 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4999999999999996e-06 2024-09-16T04:08:53.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633558+0000", 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002131796 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633663+0000", 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105833 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634106+0000", 2024-09-16T04:08:53.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00044296899999999998 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.636380+0000", 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022740849999999999 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646383+0000", 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01000238 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646457+0000", 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4258000000000006e-05 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:528 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.019845621999998, 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65958352099999995, 2024-09-16T04:08:53.692 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 528 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631479+0000", 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9599999999999998e-07 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631483+0000", 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8160000000000004e-06 2024-09-16T04:08:53.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631494+0000", 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0407999999999999e-05 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631501+0000", 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2269999999999996e-06 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634655+0000", 2024-09-16T04:08:53.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031544339999999998 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634732+0000", 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6495999999999994e-05 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646864+0000", 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012131807999999999 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646965+0000", 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010173 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.291062+0000", 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64409670600000002 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 67/56 ECSubWrite(tid=321, reqid=osd.3.0:321, at_version=67'92, trim_to=0'0, roll_forward_to=66'90))", 2024-09-16T04:08:53.697 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.533170+0000", 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.118153682999999, 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086084990000000004, 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533170+0000", 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533170+0000", 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533171+0000", 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.05e-07 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533221+0000", 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0498999999999999e-05 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533284+0000", 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2853000000000003e-05 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533293+0000", 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2529999999999993e-06 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533356+0000", 2024-09-16T04:08:53.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3107000000000004e-05 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533365+0000", 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9390000000000003e-06 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541744+0000", 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083788960000000003 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.541779+0000", 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4746999999999998e-05 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s7 67/46 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=67'20, trim_to=0'0, roll_forward_to=66'18))", 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.117710279000001, 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090747759999999997, 2024-09-16T04:08:53.702 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8199999999999999e-07 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533635+0000", 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1378000000000001e-05 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533659+0000", 2024-09-16T04:08:53.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.374e-05 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533667+0000", 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2230000000000007e-06 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533785+0000", 2024-09-16T04:08:53.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117607 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533795+0000", 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0180000000000001e-05 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542641+0000", 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088456149999999994 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542689+0000", 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7851e-05 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s2 68/56 ECSubWrite(tid=134, reqid=osd.0.0:134, at_version=68'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.109289302000001, 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010826218, 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.707 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8599999999999998e-07 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542054+0000", 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8423999999999998e-05 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542081+0000", 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7806000000000001e-05 2024-09-16T04:08:53.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542089+0000", 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5789999999999998e-06 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542147+0000", 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7978e-05 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542157+0000", 2024-09-16T04:08:53.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552766+0000", 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010609143999999999 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552861+0000", 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4841999999999998e-05 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 68/56 ECSubWrite(tid=330, reqid=osd.3.0:330, at_version=68'75, trim_to=0'0, roll_forward_to=64'73))", 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.108234346, 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094449849999999995, 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:53.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6000000000000003e-07 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543110+0000", 2024-09-16T04:08:53.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9695e-05 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543139+0000", 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8467999999999999e-05 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543148+0000", 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5349999999999993e-06 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543200+0000", 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1536999999999998e-05 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543206+0000", 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9379999999999997e-06 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552495+0000", 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.009289538 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552535+0000", 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9613999999999997e-05 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:695 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~589824] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.2022976300000003, 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014512838, 2024-09-16T04:08:53.716 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 695 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449028+0000", 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.917e-06 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450009+0000", 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000980367 2024-09-16T04:08:53.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450023+0000", 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.416e-05 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450032+0000", 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5820000000000008e-06 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450094+0000", 2024-09-16T04:08:53.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1698000000000005e-05 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450210+0000", 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116104 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451973+0000", 2024-09-16T04:08:53.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001762958 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463392+0000", 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011419024999999999 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463450+0000", 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8251999999999997e-05 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463539+0000", 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8775000000000003e-05 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s7 73/56 ECSubWrite(tid=129, reqid=osd.8.0:129, at_version=73'80, trim_to=0'0, roll_forward_to=71'78))", 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.451542+0000", 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.1997824929999998, 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086628820000000002, 2024-09-16T04:08:53.722 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451542+0000", 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451542+0000", 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451542+0000", 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.92e-07 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451563+0000", 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1348e-05 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451598+0000", 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4958e-05 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451607+0000", 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5369999999999997e-06 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451663+0000", 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6773e-05 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451673+0000", 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5570000000000001e-06 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460177+0000", 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0085042820000000002 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460204+0000", 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7234999999999999e-05 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.726 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:696 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~753664] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.199165196, 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013764281, 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:53.727 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 696 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452161+0000", 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2579999999999998e-06 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453157+0000", 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099543299999999999 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453172+0000", 2024-09-16T04:08:53.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5917e-05 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453182+0000", 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3959999999999995e-06 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455292+0000", 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021103379999999998 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455406+0000", 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000114203 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456783+0000", 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001376377 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463544+0000", 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067608640000000001 2024-09-16T04:08:53.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465825+0000", 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022816350000000002 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465923+0000", 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7860000000000002e-05 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:697 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~475136] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.456443+0000", 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.1948805680000003, 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012866471000000001, 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 697 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.733 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456443+0000", 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456443+0000", 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456445+0000", 2024-09-16T04:08:53.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6220000000000001e-06 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457074+0000", 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000629248 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457087+0000", 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2906e-05 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457096+0000", 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9449999999999999e-06 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457956+0000", 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00086007 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458062+0000", 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105559 2024-09-16T04:08:53.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.459157+0000", 2024-09-16T04:08:53.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00109566 2024-09-16T04:08:53.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463617+0000", 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044591639999999998 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469249+0000", 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056324160000000003 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469310+0000", 2024-09-16T04:08:53.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0881000000000002e-05 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:698 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~327680] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.458939+0000", 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.1923853649999998, 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010478334000000001, 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:53.739 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 698 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458939+0000", 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458939+0000", 2024-09-16T04:08:53.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458940+0000", 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5770000000000001e-06 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460359+0000", 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001418334 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460372+0000", 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3025e-05 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460379+0000", 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4680000000000003e-06 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460452+0000", 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.339e-05 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460661+0000", 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00020814299999999999 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.461550+0000", 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00088961800000000001 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465933+0000", 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043829330000000003 2024-09-16T04:08:53.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469379+0000", 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034461069999999999 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469417+0000", 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7738999999999999e-05 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.744 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:699 3.11s0 3:8bfd968c:::smithi01639990-27:head [setxattr _header (57) in=7b] snapc 18=[18,16,15,13] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.471289+0000", 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.1800349069999996, 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012440791, 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 699 2024-09-16T04:08:53.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471289+0000", 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471289+0000", 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471290+0000", 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1800000000000005e-07 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471292+0000", 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6180000000000002e-06 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471297+0000", 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7419999999999997e-06 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471302+0000", 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9320000000000004e-06 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471382+0000", 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9864000000000006e-05 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471444+0000", 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2249999999999995e-05 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471785+0000", 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000340267 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472441+0000", 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00065623500000000002 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.483660+0000", 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011219632 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.483730+0000", 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9533e-05 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:700 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:53.750 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.180014688, 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014504263, 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 700 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471310+0000", 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.08e-07 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471312+0000", 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0540000000000002e-06 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471318+0000", 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5080000000000002e-06 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471322+0000", 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3629999999999998e-06 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472045+0000", 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00072327100000000005 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472133+0000", 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8528000000000003e-05 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484130+0000", 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011996163000000001 2024-09-16T04:08:53.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484213+0000", 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3800000000000004e-05 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.485814+0000", 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016001679999999999 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.1206897930000004, 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095992542, 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530636+0000", 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.522e-06 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:53.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530651+0000", 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4980999999999999e-05 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625860+0000", 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095209590999999996 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625873+0000", 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2413e-05 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625978+0000", 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010492099999999999 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625994+0000", 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6058000000000001e-05 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626583+0000", 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058931700000000001 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:53.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:53.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:53.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626627+0000", 2024-09-16T04:08:53.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3739000000000003e-05 2024-09-16T04:08:53.760 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.760 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.760 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.760 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:53.760 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:53.760 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:53.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:53.762 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:53.760+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:53.862 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:53.861+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:53.905 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:53.905 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:53.906 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:53.906 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:53.914 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:53.963 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:53.962+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:53.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:54.016 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:54.016 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:54.016 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:54.016 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:54.016 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:54.025 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:54.065 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:54.063+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:54.129 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:124 1.0 e22/19 1:30306672:devicehealth::main.db.0000000000000000:head v 22'86, mlcod=21'84)", 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 61.696411787000002, 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020095396000000001, 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.130 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430782+0000", 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9699999999999996e-07 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430789+0000", 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4660000000000002e-06 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450028+0000", 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019238816999999998 2024-09-16T04:08:54.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450042+0000", 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4487e-05 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450169+0000", 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126787 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450188+0000", 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450816+0000", 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062741399999999997 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450877+0000", 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0782999999999997e-05 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.134 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:197 3.es0 3:7f241f48:::smithi01639990-30:head [copy-from ver 23] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.917202439, 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.023242381999999999, 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.135 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 197 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6499999999999999e-07 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209995+0000", 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.388e-06 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210002+0000", 2024-09-16T04:08:54.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5209999999999997e-06 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210008+0000", 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8780000000000003e-06 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8421999999999999e-05 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210099+0000", 2024-09-16T04:08:54.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2469999999999997e-05 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227450+0000", 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017351235 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232352+0000", 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049017239999999997 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:54.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233168+0000", 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081616600000000001 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233233+0000", 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5012999999999996e-05 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:198 3.es0 3:7f241f48:::smithi01639990-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.917172129999997, 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019673758, 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 198 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:54.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3099999999999998e-07 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210024+0000", 2024-09-16T04:08:54.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2620000000000001e-06 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210030+0000", 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9020000000000002e-06 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210037+0000", 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1480000000000003e-06 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210227+0000", 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00019055499999999999 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210269+0000", 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2246999999999997e-05 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229480+0000", 2024-09-16T04:08:54.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019210896000000002 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229587+0000", 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107026 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229695+0000", 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107291 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.145 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:213 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.011093625000001, 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53275775299999995, 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.146 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 213 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116100+0000", 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116103+0000", 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.174e-06 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116109+0000", 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4330000000000003e-06 2024-09-16T04:08:54.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116114+0000", 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1399999999999999e-06 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119298+0000", 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031846729999999998 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119365+0000", 2024-09-16T04:08:54.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6484999999999998e-05 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119675+0000", 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00031021999999999999 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122365+0000", 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002689851 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:54.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648785+0000", 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52641971300000001 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648857+0000", 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2408000000000002e-05 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:214 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:54.151 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.011070949, 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53416417900000002, 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 214 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:54.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2100000000000002e-07 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116125+0000", 2024-09-16T04:08:54.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2649999999999999e-06 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116132+0000", 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5190000000000003e-06 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116137+0000", 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9869999999999999e-06 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120036+0000", 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038984979999999998 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120118+0000", 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2312999999999996e-05 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649191+0000", 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52907320199999996 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649279+0000", 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7585999999999998e-05 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.650286+0000", 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010073879999999999 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "replica_scrub(pg: 3.4s1,from:0'0,to:37'17,epoch:47/46,start:3:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T04:08:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 31.269269412, 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16799329900000001, 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9400000000000005e-07 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857929+0000", 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4719999999999997e-06 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025793+0000", 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16786378299999999 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025803+0000", 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9629999999999993e-06 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025875+0000", 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2914e-05 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025887+0000", 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1345999999999999e-05 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025917+0000", 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0127000000000001e-05 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.4s1,RELEASE,e:47,from:1(0),reservation_nonce:0)", 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 31.100124437000002, 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26135687200000002, 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2800000000000005e-07 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027075+0000", 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3859999999999998e-06 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288302+0000", 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26122710999999998 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288308+0000", 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6189999999999999e-06 2024-09-16T04:08:54.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288392+0000", 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4549000000000007e-05 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288414+0000", 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1576000000000001e-05 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288425+0000", 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1704e-05 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:385 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.661122164999998, 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01911771, 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:54.165 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 385 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466073+0000", 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4389999999999999e-06 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466084+0000", 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1157e-05 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466101+0000", 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6334999999999999e-05 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466111+0000", 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466255+0000", 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014411 2024-09-16T04:08:54.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466263+0000", 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.858e-06 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474564+0000", 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083014019999999994 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474734+0000", 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016996599999999999 2024-09-16T04:08:54.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475786+0000", 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010515909999999999 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.482124+0000", 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063382530000000003 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485117+0000", 2024-09-16T04:08:54.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029928120000000001 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485189+0000", 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1868999999999999e-05 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:386 3.8s0 3:17b72cd6:::smithi01639990-47:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:54.171 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.661050432, 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019422101000000001, 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 386 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8100000000000002e-07 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466147+0000", 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6559999999999998e-06 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466155+0000", 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6240000000000002e-06 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466162+0000", 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4850000000000003e-06 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466343+0000", 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000181077 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466349+0000", 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1229999999999997e-06 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476407+0000", 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010057885000000001 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476516+0000", 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108738 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477098+0000", 2024-09-16T04:08:54.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000582024 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484050+0000", 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0069524950000000004 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485519+0000", 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014691280000000001 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485565+0000", 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5185000000000002e-05 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:387 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.661022331000002, 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51761721800000005, 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.178 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 387 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6399999999999998e-07 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466173+0000", 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.396e-06 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466179+0000", 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9290000000000003e-06 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466184+0000", 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9760000000000003e-06 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466397+0000", 2024-09-16T04:08:54.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021269400000000001 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466406+0000", 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0769999999999997e-06 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479302+0000", 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012896014000000001 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479415+0000", 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112767 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479914+0000", 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00049861199999999995 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484063+0000", 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004149158 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:54.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983716+0000", 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49965376500000003 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983788+0000", 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1465999999999996e-05 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/46 ECSubWriteReply(tid=59, last_complete=48'38, committed=1, applied=1))", 2024-09-16T04:08:54.184 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.642981319, 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50006588100000005, 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2499999999999999e-07 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484214+0000", 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6440000000000001e-06 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983573+0000", 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49935939200000001 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983581+0000", 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983628+0000", 2024-09-16T04:08:54.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7642999999999998e-05 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.984278+0000", 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00064939900000000005 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "rep_scrubmap(3.8s0 e46 from shard 10(6))", 2024-09-16T04:08:54.188 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.140880027000001, 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21666559099999999, 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986314+0000", 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5600000000000005e-07 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986319+0000", 2024-09-16T04:08:54.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0950000000000003e-06 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202862+0000", 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21654300900000001 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202872+0000", 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0077e-05 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202931+0000", 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9568000000000003e-05 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202942+0000", 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0139999999999999e-05 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202979+0000", 2024-09-16T04:08:54.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6946000000000002e-05 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s3 53/48 ECSubWrite(tid=39, reqid=osd.4.0:39, at_version=53'24, trim_to=0'0, roll_forward_to=51'22))", 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.638255238999999, 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017845360000000001, 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.193 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2000000000000001e-07 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488964+0000", 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6418e-05 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502624+0000", 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013659439000000001 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502636+0000", 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2401999999999999e-05 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502702+0000", 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6006999999999993e-05 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502712+0000", 2024-09-16T04:08:54.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8870000000000004e-06 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506740+0000", 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040279319999999997 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506783+0000", 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3055e-05 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:446 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.623653579999999, 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67862900900000001, 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.198 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 446 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503541+0000", 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3680000000000001e-06 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503545+0000", 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.561e-06 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503553+0000", 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8939999999999993e-06 2024-09-16T04:08:54.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503559+0000", 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.224e-06 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505239+0000", 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001679739 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505303+0000", 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4219000000000003e-05 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505673+0000", 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000369374 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.507082+0000", 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014090330000000001 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182055+0000", 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67497357700000005 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182168+0000", 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011302 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.203 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:447 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.623624073999999, 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.680085987, 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 447 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503570+0000", 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.44e-07 2024-09-16T04:08:54.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503576+0000", 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4559999999999997e-06 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503602+0000", 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6264000000000001e-05 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503610+0000", 2024-09-16T04:08:54.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4109999999999999e-06 2024-09-16T04:08:54.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505945+0000", 2024-09-16T04:08:54.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023353390000000001 2024-09-16T04:08:54.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506004+0000", 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9410999999999998e-05 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182603+0000", 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67659811199999997 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182695+0000", 2024-09-16T04:08:54.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1951000000000001e-05 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.183655+0000", 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00096039900000000002 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:54.209 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.618551749000002, 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.299096274, 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:54.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508642+0000", 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5400000000000002e-07 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508645+0000", 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9249999999999999e-06 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807520+0000", 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.29887539899999999 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807534+0000", 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3643e-05 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807681+0000", 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014696699999999999 2024-09-16T04:08:54.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807738+0000", 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6885999999999997e-05 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.618170284000001, 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67354323000000005, 2024-09-16T04:08:54.213 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:54.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9500000000000002e-07 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509027+0000", 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4429999999999999e-06 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181881+0000", 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67285377899999999 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181891+0000", 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0479000000000001e-05 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181961+0000", 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0370000000000006e-05 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182566+0000", 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060456400000000003 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.4s1 62/56 ECSubRead(tid=8, to_read={3:26eb5942:::smithi01639990-23:head=0,4096,0}, subchunks={3:26eb5942:::smithi01639990-23:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.479539454000001, 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64312231099999995, 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.217 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2500000000000004e-07 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647678+0000", 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3421000000000002e-05 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290437+0000", 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64275884900000002 2024-09-16T04:08:54.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290450+0000", 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2858e-05 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290545+0000", 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5463000000000004e-05 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290776+0000", 2024-09-16T04:08:54.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000230895 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:705 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.6344530050000001, 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.037755205, 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:54.221 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 705 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:08:54.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492741+0000", 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6199999999999997e-07 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492743+0000", 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7530000000000002e-06 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492750+0000", 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7710000000000001e-06 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492756+0000", 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.524e-06 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494038+0000", 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012823310000000001 2024-09-16T04:08:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494140+0000", 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010216400000000001 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494664+0000", 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052402199999999996 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495429+0000", 2024-09-16T04:08:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00076470399999999995 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530435+0000", 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035005738000000002 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530495+0000", 2024-09-16T04:08:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0535999999999998e-05 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:706 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.6344272809999998, 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.039505024, 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 706 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0600000000000001e-07 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492769+0000", 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7420000000000002e-06 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492776+0000", 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0480000000000001e-06 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492781+0000", 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1499999999999998e-06 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495503+0000", 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027220199999999999 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495628+0000", 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000125041 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530799+0000", 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035171257999999997 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530897+0000", 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7608999999999993e-05 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.532271+0000", 2024-09-16T04:08:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00137375 2024-09-16T04:08:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:54.232 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:54.233 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:54.234 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:54.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:54.235 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:54.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:54.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:54.237 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:54.237 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:54.231+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:54.237 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:54.238 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:54.259 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:54.259 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:54.281 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:54.281 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:54.302 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:54.324 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:54.324 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:54.334 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:54.332+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:54.345 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:54.345 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:54.346 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:54.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:54.367 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:54.367 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:54.435 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:54.434+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:54.465 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-16T04:08:54.467 INFO:tasks.rados.rados.0.smithi016.stdout:308: done (0 left) 2024-09-16T04:08:54.467 INFO:tasks.rados.rados.0.smithi016.stdout:309: setattr oid 37 current snap is 27 2024-09-16T04:08:54.470 INFO:tasks.rados.rados.0.smithi016.stdout:310: copy_from oid 34 from oid 33 current snap is 27 2024-09-16T04:08:54.470 INFO:tasks.rados.rados.0.smithi016.stdout:311: copy_from oid 16 from oid 43 current snap is 27 2024-09-16T04:08:54.470 INFO:tasks.rados.rados.0.smithi016.stdout:312: snap_remove snap 22 2024-09-16T04:08:54.477 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:54.478 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:54.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:54.537 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:54.535+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:54.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:08:54.637 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:54.637 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:54.643 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:54.642+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:54.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:08:54.703 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:54.704 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:54.713 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-16T04:08:54.738 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:54.736+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:54.795 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:54.795 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:54.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:08:54.839 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:54.838+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:08:54.911 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:54.911 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:54.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:08:54.964 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:54.962+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:55.028 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:55.028 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:55.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:08:55.041 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:55.039+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:55.142 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:55.140+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:55.184 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:55.184 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:55.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:08:55.243 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:55.241+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:55.300 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:55.300 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:55.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:08:55.355 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:55.353+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:55.417 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:55.417 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:55.425 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:08:55.437 INFO:tasks.rados.rados.0.smithi016.stdout:312: done (3 left) 2024-09-16T04:08:55.437 INFO:tasks.rados.rados.0.smithi016.stdout:313: rollback oid 35 current snap is 27 2024-09-16T04:08:55.437 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 37 v 105 (ObjNum 161 snap 0 seq_num 3714699099) dirty exists 2024-09-16T04:08:55.437 INFO:tasks.rados.rados.0.smithi016.stdout:311: finishing copy_from racing read to smithi01639990-16 2024-09-16T04:08:55.437 INFO:tasks.rados.rados.0.smithi016.stdout:310: finishing copy_from to smithi01639990-34 2024-09-16T04:08:55.437 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 34 v 8 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T04:08:55.438 INFO:tasks.rados.rados.0.smithi016.stdout:310: finishing copy_from racing read to smithi01639990-34 2024-09-16T04:08:55.438 INFO:tasks.rados.rados.0.smithi016.stdout:311: finishing copy_from to smithi01639990-16 2024-09-16T04:08:55.438 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 16 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T04:08:55.438 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 35 to 20 2024-09-16T04:08:55.438 INFO:tasks.rados.rados.0.smithi016.stdout:309: done (3 left) 2024-09-16T04:08:55.438 INFO:tasks.rados.rados.0.smithi016.stdout:310: done (2 left) 2024-09-16T04:08:55.438 INFO:tasks.rados.rados.0.smithi016.stdout:311: done (1 left) 2024-09-16T04:08:55.438 INFO:tasks.rados.rados.0.smithi016.stdout:314: read oid 49 snap 20 2024-09-16T04:08:55.438 INFO:tasks.rados.rados.0.smithi016.stdout:314: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-16T04:08:55.439 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 43 current snap is 27 2024-09-16T04:08:55.439 INFO:tasks.rados.rados.0.smithi016.stdout:315: seq_num 132 ranges {720896=671744,1392640=573440,1966080=753664,2719744=229376} 2024-09-16T04:08:55.445 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:55.443+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:55.445 INFO:tasks.rados.rados.0.smithi016.stdout:315: writing smithi01639990-43 from 720896 to 1392640 tid 1 2024-09-16T04:08:55.449 INFO:tasks.rados.rados.0.smithi016.stdout:315: writing smithi01639990-43 from 1392640 to 1966080 tid 2 2024-09-16T04:08:55.455 INFO:tasks.rados.rados.0.smithi016.stdout:315: writing smithi01639990-43 from 1966080 to 2719744 tid 3 2024-09-16T04:08:55.455 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:55.453+0000 7f304f68e640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 18 after trimming was started 2024-09-16T04:08:55.456 INFO:tasks.rados.rados.0.smithi016.stdout:315: writing smithi01639990-43 from 2719744 to 2949120 tid 4 2024-09-16T04:08:55.456 INFO:tasks.rados.rados.0.smithi016.stdout:316: delete oid 33 current snap is 27 2024-09-16T04:08:55.457 INFO:tasks.rados.rados.0.smithi016.stdout:314: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-16T04:08:55.534 INFO:tasks.rados.rados.0.smithi016.stdout:315: finishing write tid 1 to smithi01639990-43 2024-09-16T04:08:55.534 INFO:tasks.rados.rados.0.smithi016.stdout:315: finishing write tid 2 to smithi01639990-43 2024-09-16T04:08:55.534 INFO:tasks.rados.rados.0.smithi016.stdout:315: finishing write tid 3 to smithi01639990-43 2024-09-16T04:08:55.534 INFO:tasks.rados.rados.0.smithi016.stdout:313: finishing rollback tid 0 to smithi01639990-35 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:313: finishing rollback tid 1 to smithi01639990-35 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:315: finishing write tid 4 to smithi01639990-43 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:315: finishing write tid 5 to smithi01639990-43 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:313: finishing rollback tid 2 to smithi01639990-35 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 35 v 34 (ObjNum 69 snap 3 seq_num 69) dirty exists 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:315: finishing write tid 6 to smithi01639990-43 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:315: oid 43 updating version 0 to 107 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:315: oid 43 updating version 107 to 111 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:315: oid 43 version 111 is already newer than 110 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:315: oid 43 version 111 is already newer than 108 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:315: oid 43 version 111 is already newer than 109 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 43 v 111 (ObjNum 132 snap 27 seq_num 132) dirty exists 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:315: left oid 43 (ObjNum 132 snap 27 seq_num 132) 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:313: done (3 left) 2024-09-16T04:08:55.535 INFO:tasks.rados.rados.0.smithi016.stdout:314: done (2 left) 2024-09-16T04:08:55.536 INFO:tasks.rados.rados.0.smithi016.stdout:315: done (1 left) 2024-09-16T04:08:55.536 INFO:tasks.rados.rados.0.smithi016.stdout:316: done (0 left) 2024-09-16T04:08:55.536 INFO:tasks.rados.rados.0.smithi016.stdout:317: copy_from oid 38 from oid 41 current snap is 27 2024-09-16T04:08:55.536 INFO:tasks.rados.rados.0.smithi016.stdout:318: copy_from oid 46 from oid 31 current snap is 27 2024-09-16T04:08:55.536 INFO:tasks.rados.rados.0.smithi016.stdout:319: snap_remove snap 24 2024-09-16T04:08:55.546 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:55.545+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:55.559 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:55.559 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:55.568 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:08:55.652 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:55.651+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:55.714 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:55.714 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:55.722 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:08:55.747 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:55.745+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:55.848 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:55.846+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:55.866 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:55.866 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:55.875 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:08:55.949 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:55.947+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:56.027 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:08:56.050 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:56.049+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:56.130 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:56.130 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:56.140 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:08:56.183 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:56.182+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:56.243 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:56.243 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:56.252 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:08:56.252 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:08:56.250+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:08:56.353 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:56.350+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:56.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:08:56.394 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:56.394 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:56.403 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:08:56.453 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:56.451+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:56.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:08:56.484 INFO:tasks.rados.rados.0.smithi016.stdout:319: done (2 left) 2024-09-16T04:08:56.484 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 29 current snap is 27 2024-09-16T04:08:56.484 INFO:tasks.rados.rados.0.smithi016.stdout:318: finishing copy_from racing read to smithi01639990-46 2024-09-16T04:08:56.484 INFO:tasks.rados.rados.0.smithi016.stdout:320: seq_num 134 ranges {393216=704512,1097728=475136,1572864=573440,2146304=507904} 2024-09-16T04:08:56.491 INFO:tasks.rados.rados.0.smithi016.stdout:320: writing smithi01639990-29 from 393216 to 1097728 tid 1 2024-09-16T04:08:56.494 INFO:tasks.rados.rados.0.smithi016.stdout:320: writing smithi01639990-29 from 1097728 to 1572864 tid 2 2024-09-16T04:08:56.499 INFO:tasks.rados.rados.0.smithi016.stdout:320: writing smithi01639990-29 from 1572864 to 2146304 tid 3 2024-09-16T04:08:56.502 INFO:tasks.rados.rados.0.smithi016.stdout:320: writing smithi01639990-29 from 2146304 to 2654208 tid 4 2024-09-16T04:08:56.503 INFO:tasks.rados.rados.0.smithi016.stdout:321: snap_create 2024-09-16T04:08:56.503 INFO:tasks.rados.rados.0.smithi016.stdout:317: finishing copy_from racing read to smithi01639990-38 2024-09-16T04:08:56.503 INFO:tasks.rados.rados.0.smithi016.stdout:318: finishing copy_from to smithi01639990-46 2024-09-16T04:08:56.503 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 46 v 18 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T04:08:56.503 INFO:tasks.rados.rados.0.smithi016.stdout:317: finishing copy_from to smithi01639990-38 2024-09-16T04:08:56.503 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 38 v 58 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-09-16T04:08:56.506 INFO:tasks.rados.rados.0.smithi016.stdout:320: finishing write tid 1 to smithi01639990-29 2024-09-16T04:08:56.506 INFO:tasks.rados.rados.0.smithi016.stdout:320: finishing write tid 2 to smithi01639990-29 2024-09-16T04:08:56.507 INFO:tasks.rados.rados.0.smithi016.stdout:320: finishing write tid 3 to smithi01639990-29 2024-09-16T04:08:56.514 INFO:tasks.rados.rados.0.smithi016.stdout:320: finishing write tid 4 to smithi01639990-29 2024-09-16T04:08:56.514 INFO:tasks.rados.rados.0.smithi016.stdout:320: finishing write tid 5 to smithi01639990-29 2024-09-16T04:08:56.516 INFO:tasks.rados.rados.0.smithi016.stdout:320: finishing write tid 6 to smithi01639990-29 2024-09-16T04:08:56.516 INFO:tasks.rados.rados.0.smithi016.stdout:320: oid 29 updating version 0 to 108 2024-09-16T04:08:56.516 INFO:tasks.rados.rados.0.smithi016.stdout:320: oid 29 version 108 is already newer than 106 2024-09-16T04:08:56.516 INFO:tasks.rados.rados.0.smithi016.stdout:320: oid 29 version 108 is already newer than 104 2024-09-16T04:08:56.517 INFO:tasks.rados.rados.0.smithi016.stdout:320: oid 29 version 108 is already newer than 105 2024-09-16T04:08:56.517 INFO:tasks.rados.rados.0.smithi016.stdout:320: oid 29 version 108 is already newer than 107 2024-09-16T04:08:56.517 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 29 v 108 (ObjNum 134 snap 27 seq_num 134) dirty exists 2024-09-16T04:08:56.517 INFO:tasks.rados.rados.0.smithi016.stdout:320: left oid 29 (ObjNum 134 snap 27 seq_num 134) 2024-09-16T04:08:56.522 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:56.522 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:56.532 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:08:56.541 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:56.541 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:56.553 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:56.552+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:56.564 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:56.564 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:56.585 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:56.585 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:56.607 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:56.640 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:56.640 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:56.655 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:56.653+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:56.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:56.678 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:56.679 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:56.688 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:08:56.698 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:56.698 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:56.736 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:56.736 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:56.755 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:56.753+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:08:56.757 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:56.797 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:56.797 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:56.830 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:56.831 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:56.831 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:08:56.831 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:56.837 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:56.837 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:56.839 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:08:56.856 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:08:56.855+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:08:56.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:56.877 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:56.942 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:56.942 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:08:56.942 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:08:56.942 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:08:56.942 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:56.950 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:08:56.957 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:56.955+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:08:57.056 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:08:57.056 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:08:57.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:08:57.056 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:08:57.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 67.111134738000004, 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.057 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:08:57.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:08:57.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:08:57.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.063 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 66.128730494999999, 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:57.064 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:08:57.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:57.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.069 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 66.118367989000006, 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:08:57.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 64.621779936999999, 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.074 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:08:57.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:08:57.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:08:57.079 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:164 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 64.167205444000004, 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011745096, 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.080 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 164 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884652+0000", 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884688+0000", 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5243999999999997e-05 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884692+0000", 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5839999999999996e-06 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884701+0000", 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6819999999999992e-06 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893436+0000", 2024-09-16T04:08:57.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087352260000000004 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893480+0000", 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3948e-05 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.893550+0000", 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0149999999999998e-05 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894687+0000", 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00113722 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896300+0000", 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016123680000000001 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896357+0000", 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7025999999999999e-05 2024-09-16T04:08:57.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896397+0000", 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0411999999999999e-05 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 64.167151383000004, 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.086 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:57.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:08:57.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:08:57.091 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.092 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.942882845, 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:08:57.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:08:57.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:08:57.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:08:57.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.364948884999997, 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.098 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:08:57.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:08:57.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:08:57.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:08:57.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:08:57.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:08:57.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.102 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.102 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:08:57.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:08:57.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:08:57.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:08:57.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:08:57.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:08:57.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:08:57.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:08:57.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:57.106 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 47.948135739999998, 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:57.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.108 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.108 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:08:57.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:08:57.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:08:57.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:08:57.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:08:57.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:08:57.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:08:57.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:08:57.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:08:57.112 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 47.946748325999998, 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:08:57.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:08:57.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.117 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.117 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.117 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:08:57.117 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:08:57.117 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.117 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.117 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=188, reqid=client.4440.0:379, at_version=47'56, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.599616869000002, 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011758437, 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.118 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.119 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452241+0000", 2024-09-16T04:08:57.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9699999999999995e-07 2024-09-16T04:08:57.120 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.120 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.120 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.120 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.452266+0000", 2024-09-16T04:08:57.120 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4151e-05 2024-09-16T04:08:57.121 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.121 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.121 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.121 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459768+0000", 2024-09-16T04:08:57.121 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075025179999999997 2024-09-16T04:08:57.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.122 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.122 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.122 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459778+0000", 2024-09-16T04:08:57.122 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0244999999999999e-05 2024-09-16T04:08:57.122 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.123 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.123 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.123 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459835+0000", 2024-09-16T04:08:57.123 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6394999999999998e-05 2024-09-16T04:08:57.123 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.124 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.124 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.124 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.459845+0000", 2024-09-16T04:08:57.124 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0518e-05 2024-09-16T04:08:57.124 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.125 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.125 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463963+0000", 2024-09-16T04:08:57.125 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041182450000000004 2024-09-16T04:08:57.125 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.125 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.126 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.126 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.463999+0000", 2024-09-16T04:08:57.126 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5667999999999998e-05 2024-09-16T04:08:57.126 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.126 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.127 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.127 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.127 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:08:57.127 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:57.127 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.587429296000003, 2024-09-16T04:08:57.128 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:08:57.128 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.128 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.128 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.128 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.129 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.129 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:57.129 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.129 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.130 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.130 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:08:57.130 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.130 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.131 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.131 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:08:57.131 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:08:57.131 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.131 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.132 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.132 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:08:57.132 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:08:57.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.132 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:08:57.134 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:57.135 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.614559346, 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019153804, 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.136 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437299+0000", 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.137 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437320+0000", 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1121999999999999e-05 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437388+0000", 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7889e-05 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437397+0000", 2024-09-16T04:08:57.138 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4390000000000006e-06 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441658+0000", 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042607110000000004 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441667+0000", 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.139 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456416+0000", 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014748493 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456452+0000", 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6159999999999999e-05 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:57.140 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.611369634999999, 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017174697999999999, 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.141 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440489+0000", 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.015e-06 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440509+0000", 2024-09-16T04:08:57.142 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0338000000000001e-05 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440547+0000", 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440557+0000", 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7820000000000005e-06 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.143 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442068+0000", 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015112179999999999 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442078+0000", 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4620000000000002e-06 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457639+0000", 2024-09-16T04:08:57.144 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015560954 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457663+0000", 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4121000000000002e-05 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.as7 68/56 ECSubWrite(tid=57, reqid=osd.6.0:57, at_version=68'42, trim_to=0'0, roll_forward_to=64'40))", 2024-09-16T04:08:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.517702383, 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017528866000000001, 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.49e-07 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534179+0000", 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3308000000000001e-05 2024-09-16T04:08:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546651+0000", 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012471873 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546660+0000", 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2229999999999996e-06 2024-09-16T04:08:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546730+0000", 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0123000000000006e-05 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546741+0000", 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0684e-05 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.149 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551646+0000", 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049059180000000004 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551684+0000", 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7588e-05 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 68/56 ECSubWriteReply(tid=200, last_complete=68'24, committed=1, applied=1))", 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 13.488690359, 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46574167999999999, 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563168+0000", 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4099999999999997e-07 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563174+0000", 2024-09-16T04:08:57.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.327e-06 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028683+0000", 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46550948399999997 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028698+0000", 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4576000000000001e-05 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028858+0000", 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016062100000000001 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028909+0000", 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0730999999999999e-05 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.154 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:654 3.ds0 3:bc502819:::smithi01639990-5:head [create,setxattr XXWdXV_kiPaajZX\\PgmqNaZqSVmab^hrtRYeagZr (1894) in=40b,setxattr NdZr`e\\^kXiksh]\\XocenQrhYc]c`[proOfo`Pd^ (33) in=40b,setxattr _`NqSdldlQbUNl^ZNRba\\naZhhdraiOgpafrZb[o (103) in=40b,setxattr iadimrq_[jOPUblhnVqTYqQNedg (1647) in=27b,setxattr `e\\rtQi]nmoP\\cm]oRYN`[]^qVUTsU^_oXm]h\\ec (6452) in=40b,setxattr ppZXc_dYi_WWXcUdeqcpck_ceUVa^XftgnfgdP[q (3576) in=40b,setxattr hehZsP\\ajqhe^nfrZjQmsnfWoqq\\e\\NV^rOaZYtj (1787) in=40b,setxattr a]Smbg`^k`emWabQRYcpr_lQj`eh`[r\\riqa_sfd (1015) in=40b,setxattr eqsaZkZ[`t^R^grhYOVrVsspVQQp_Zo\\loUV[\\U[ (982) in=40b,setxattr Rn`k`cSYQj`XOZdn_mSPrqQtoUpWTk[^[t (1647) in=34b,setxattr ]TtVabQgUTabfmqae[UVe`cNsqqYfdh]\\sVoeXQr (383) in=40b,setxattr iRn][N_NY\\ZhmYdQ[[ptTP_NW^Z]_roZnOpcmR_Y (19016) in=40b,setxattr T_b_WhnZnkaaYkNmmOUnaZa`dhRhQSshYQVQnVql (13472) in=40b,setxattr \\hZXh`dT^W[c\\XodZZ_d\\QO\\\\[PVWthO_WVOPSQd (1681) in=40b,setxattr ctSOqNY_dUqTOjgXYjqgslgNRRPoT\\\\OXPPWrtt_ (522) in=40b,setxattr f\\hTOf^sbRaYeUntSV^WhUo]qV`bSNermptfXs]Q (1645) in=40b,setxattr ohNTdfibmYs`eUTiQa^bcYW\\aarhT]YQ_haagqpf (1119) in=40b,setxattr qqUtZZ]ataUj_[pnh\\jVtU_dRpi^g[VPNeS`Y\\hk (699) in=40b,setxattr qsgedRfqdW]]cYbWOUgh^ZleaSQiteejr^afXbUj (525) in=40b,setxattr [XmPqpoSg`fWjdlmQ_qabof]_WVcPaY[]k[ardgj (594) in=40b] snapc 15=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.517679323999999, 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019213674, 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:57.155 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 654 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534181+0000", 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5830000000000001e-06 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534398+0000", 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000216663 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534421+0000", 2024-09-16T04:08:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3714e-05 2024-09-16T04:08:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534434+0000", 2024-09-16T04:08:57.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3185e-05 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534504+0000", 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9665000000000005e-05 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534510+0000", 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1709999999999999e-06 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.538902+0000", 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043914660000000001 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539044+0000", 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014215000000000001 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540400+0000", 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013559189999999999 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541692+0000", 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001291964 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553277+0000", 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011585432999999999 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553392+0000", 2024-09-16T04:08:57.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011476099999999999 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:733 3.fs0 3:f7af954f:::smithi01639990-10:head [append 0~0] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.6130476749999998, 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016112728999999999, 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.162 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 733 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:08:57.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438813+0000", 2024-09-16T04:08:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6089999999999999e-06 2024-09-16T04:08:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438822+0000", 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6199999999999994e-06 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438844+0000", 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1933000000000002e-05 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438856+0000", 2024-09-16T04:08:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1588e-05 2024-09-16T04:08:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438966+0000", 2024-09-16T04:08:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011003899999999999 2024-09-16T04:08:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:08:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438974+0000", 2024-09-16T04:08:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5019999999999992e-06 2024-09-16T04:08:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.446497+0000", 2024-09-16T04:08:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075231389999999999 2024-09-16T04:08:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.446592+0000", 2024-09-16T04:08:57.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4358999999999996e-05 2024-09-16T04:08:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.447539+0000", 2024-09-16T04:08:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00094693900000000005 2024-09-16T04:08:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454766+0000", 2024-09-16T04:08:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0072270520000000003 2024-09-16T04:08:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454838+0000", 2024-09-16T04:08:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1970000000000004e-05 2024-09-16T04:08:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454923+0000", 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4979000000000001e-05 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:734 3.fs0 3:f7af954f:::smithi01639990-10:head [rollback 1a] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.6129736220000002, 2024-09-16T04:08:57.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019165187, 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 734 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:08:57.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7399999999999998e-07 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438890+0000", 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8889999999999999e-06 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438903+0000", 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2466000000000001e-05 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438909+0000", 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3640000000000004e-06 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439564+0000", 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000655084 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439571+0000", 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.472e-06 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448110+0000", 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085393550000000002 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448181+0000", 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1259000000000003e-05 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448725+0000", 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00054394000000000003 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.457803+0000", 2024-09-16T04:08:57.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0090773959999999997 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.457955+0000", 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015255800000000001 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458049+0000", 2024-09-16T04:08:57.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3930000000000004e-05 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:735 3.fs0 3:f7af954f:::smithi01639990-10:head [append 0~0] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.6122715339999996, 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019107652999999999, 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:08:57.178 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 735 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439587+0000", 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439591+0000", 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2570000000000001e-06 2024-09-16T04:08:57.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439607+0000", 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5846e-05 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439614+0000", 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6579999999999996e-06 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439657+0000", 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3402999999999998e-05 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439663+0000", 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0959999999999997e-06 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449536+0000", 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098724250000000006 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449600+0000", 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4857999999999995e-05 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449969+0000", 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036863400000000001 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458052+0000", 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0080833569999999993 2024-09-16T04:08:57.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458113+0000", 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0764999999999998e-05 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458694+0000", 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00058057699999999998 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:08:57.184 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:08:57.185 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:08:57.186 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:08:57.187 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:57.184+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:08:57.287 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:57.285+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:08:57.301 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:57.301 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:57.301 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:57.301 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:57.310 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:08:57.388 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:57.386+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:57.413 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:57.414 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:57.414 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:57.414 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:57.414 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:57.422 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:08:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-16T04:08:57.488 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:57.487+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:57.489 INFO:tasks.rados.rados.0.smithi016.stdout:317: done (3 left) 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:318: done (2 left) 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:320: done (1 left) 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:321: done (0 left) 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:322: read oid 43 snap -1 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:322: expect (ObjNum 132 snap 27 seq_num 132) 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:323: copy_from oid 42 from oid 19 current snap is 28 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:324: read oid 20 snap -1 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:324: expect deleted 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:325: rmattr oid 7 current snap is 28 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:326: read oid 44 snap -1 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:326: expect (ObjNum 109 snap 18 seq_num 109) 2024-09-16T04:08:57.490 INFO:tasks.rados.rados.0.smithi016.stdout:327: delete oid 29 current snap is 28 2024-09-16T04:08:57.503 INFO:tasks.rados.rados.0.smithi016.stdout:324: done (5 left) 2024-09-16T04:08:57.504 INFO:tasks.rados.rados.0.smithi016.stdout:327: done (4 left) 2024-09-16T04:08:57.504 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 25 current snap is 28 2024-09-16T04:08:57.504 INFO:tasks.rados.rados.0.smithi016.stdout:328: seq_num 136 ranges {3915776=737280,4653056=770048,5423104=540672,5963776=245760} 2024-09-16T04:08:57.530 INFO:tasks.rados.rados.0.smithi016.stdout:328: writing smithi01639990-25 from 3915776 to 4653056 tid 1 2024-09-16T04:08:57.534 INFO:tasks.rados.rados.0.smithi016.stdout:328: writing smithi01639990-25 from 4653056 to 5423104 tid 2 2024-09-16T04:08:57.537 INFO:tasks.rados.rados.0.smithi016.stdout:328: writing smithi01639990-25 from 5423104 to 5963776 tid 3 2024-09-16T04:08:57.540 INFO:tasks.rados.rados.0.smithi016.stdout:328: writing smithi01639990-25 from 5963776 to 6209536 tid 4 2024-09-16T04:08:57.540 INFO:tasks.rados.rados.0.smithi016.stdout:329: read oid 46 snap -1 2024-09-16T04:08:57.540 INFO:tasks.rados.rados.0.smithi016.stdout:329: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T04:08:57.540 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 23 current snap is 28 2024-09-16T04:08:57.541 INFO:tasks.rados.rados.0.smithi016.stdout:330: seq_num 137 ranges {1753088=753664,2506752=786432,3293184=655360,3948544=114688} 2024-09-16T04:08:57.553 INFO:tasks.rados.rados.0.smithi016.stdout:330: writing smithi01639990-23 from 1753088 to 2506752 tid 1 2024-09-16T04:08:57.558 INFO:tasks.rados.rados.0.smithi016.stdout:330: writing smithi01639990-23 from 2506752 to 3293184 tid 2 2024-09-16T04:08:57.562 INFO:tasks.rados.rados.0.smithi016.stdout:330: writing smithi01639990-23 from 3293184 to 3948544 tid 3 2024-09-16T04:08:57.563 INFO:tasks.rados.rados.0.smithi016.stdout:330: writing smithi01639990-23 from 3948544 to 4063232 tid 4 2024-09-16T04:08:57.563 INFO:tasks.rados.rados.0.smithi016.stdout:331: read oid 12 snap -1 2024-09-16T04:08:57.563 INFO:tasks.rados.rados.0.smithi016.stdout:331: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:08:57.563 INFO:tasks.rados.rados.0.smithi016.stdout:332: copy_from oid 26 from oid 9 current snap is 28 2024-09-16T04:08:57.563 INFO:tasks.rados.rados.0.smithi016.stdout:323: finishing copy_from racing read to smithi01639990-42 2024-09-16T04:08:57.563 INFO:tasks.rados.rados.0.smithi016.stdout:326: expect (ObjNum 109 snap 18 seq_num 109) 2024-09-16T04:08:57.584 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:57.584 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 48.439484645, 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.585 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.586 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:08:57.587 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:08:57.588 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:57.589 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 40.121429036999999, 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:08:57.590 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.591 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.592 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.593 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 39.990886877999998, 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:57.594 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:08:57.595 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.596 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.597 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.909964324999997, 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:57.598 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:08:57.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.749560267, 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.602 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:08:57.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.073455429999999, 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.606 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:08:57.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.051742404999999, 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.612 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:08:57.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.051690178000001, 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:08:57.617 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:08:57.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 67/56 ECSubWrite(tid=320, reqid=osd.3.0:320, at_version=67'62, trim_to=0'0, roll_forward_to=66'60))", 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.048629203999999, 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01263539, 2024-09-16T04:08:57.622 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532549+0000", 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6899999999999999e-07 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532569+0000", 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0305999999999998e-05 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532682+0000", 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112361 2024-09-16T04:08:57.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532692+0000", 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0832000000000001e-05 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532753+0000", 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0568e-05 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.532763+0000", 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6010000000000001e-06 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545152+0000", 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012389908 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.545184+0000", 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1644999999999999e-05 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.11s6 68/56 ECSubWrite(tid=17, reqid=osd.10.0:17, at_version=68'19, trim_to=0'0, roll_forward_to=64'17))", 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.042004256, 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014242092, 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8500000000000001e-07 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539199+0000", 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4397999999999999e-05 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539248+0000", 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9033000000000003e-05 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539258+0000", 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8800000000000003e-06 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539858+0000", 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060046199999999996 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539867+0000", 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.363e-06 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553388+0000", 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013520249 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553416+0000", 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8121999999999999e-05 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 68/56 ECSubWrite(tid=327, reqid=osd.3.0:327, at_version=68'66, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.041712798000001, 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012797307000000001, 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.631 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539466+0000", 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4000000000000001e-07 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539484+0000", 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8391999999999999e-05 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539516+0000", 2024-09-16T04:08:57.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1439999999999997e-05 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539524+0000", 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4570000000000005e-06 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542244+0000", 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027199070000000001 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542254+0000", 2024-09-16T04:08:57.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8500000000000006e-06 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552231+0000", 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0099774170000000006 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552262+0000", 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1204000000000001e-05 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.635 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=64, reqid=osd.4.0:64, at_version=68'43, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.040890252000001, 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014089379000000001, 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:57.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1500000000000001e-07 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540308+0000", 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0585999999999999e-05 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540337+0000", 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8560000000000001e-05 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540345+0000", 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6299999999999998e-06 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540400+0000", 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4818000000000003e-05 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540411+0000", 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1009000000000001e-05 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554349+0000", 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013938394 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554377+0000", 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8167e-05 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=66, reqid=osd.4.0:66, at_version=68'45, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.039048961000001, 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01391278, 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.640 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542130+0000", 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6600000000000005e-07 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542154+0000", 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3862999999999998e-05 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542183+0000", 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9108999999999999e-05 2024-09-16T04:08:57.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542192+0000", 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549962+0000", 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007769977 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549972+0000", 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0074e-05 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556017+0000", 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060444319999999998 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556042+0000", 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5012999999999999e-05 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.644 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s7 68/56 ECSubWrite(tid=197, reqid=osd.7.0:197, at_version=68'24, trim_to=0'0, roll_forward_to=64'22))", 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.038705525999999, 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013604978, 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542473+0000", 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5099999999999999e-07 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542491+0000", 2024-09-16T04:08:57.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8417e-05 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542561+0000", 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9744000000000002e-05 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542568+0000", 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.289e-06 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546547+0000", 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039792350000000002 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546557+0000", 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8870000000000004e-06 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556052+0000", 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094947169999999997 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556077+0000", 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5537999999999998e-05 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=119, reqid=client.4440.0:705, at_version=73'60, trim_to=0'0, roll_forward_to=73'57))", 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.0864928860000003, 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035584957, 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494686+0000", 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9999999999999996e-07 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494705+0000", 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9208999999999999e-05 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529770+0000", 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035064924999999997 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529777+0000", 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5069999999999997e-06 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529831+0000", 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4122000000000001e-05 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529845+0000", 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4571e-05 2024-09-16T04:08:57.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530243+0000", 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039740100000000003 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530270+0000", 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7421999999999999e-05 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:708 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 69] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.0694233460000007, 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49730772000000001, 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 708 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511756+0000", 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6029999999999999e-06 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511762+0000", 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2500000000000003e-06 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511774+0000", 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1761e-05 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511797+0000", 2024-09-16T04:08:57.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2416999999999999e-05 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511872+0000", 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5437000000000003e-05 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511975+0000", 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010341899999999999 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.526578+0000", 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014602084 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529422+0000", 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028444880000000001 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008984+0000", 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47956190999999998 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009062+0000", 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8350999999999996e-05 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:709 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.0693678440000003, 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017140263999999999, 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.659 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 709 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:08:57.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0800000000000001e-07 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511812+0000", 2024-09-16T04:08:57.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7799999999999999e-06 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511816+0000", 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.172e-06 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511819+0000", 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9739999999999998e-06 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512187+0000", 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036796799999999998 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512240+0000", 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3013999999999998e-05 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528792+0000", 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016551461999999999 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528863+0000", 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0797000000000007e-05 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528950+0000", 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7788999999999996e-05 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:08:57.664 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.0505513279999992, 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.36295003300000001, 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7899999999999998e-07 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530637+0000", 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892881+0000", 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.362243749 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892890+0000", 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5620000000000004e-06 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892950+0000", 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0270999999999999e-05 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892959+0000", 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1689999999999998e-06 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893539+0000", 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000579628 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893577+0000", 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8056999999999998e-05 2024-09-16T04:08:57.668 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=131, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 9.0502667910000003, 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47861527599999998, 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530912+0000", 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4400000000000001e-07 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530915+0000", 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5530000000000001e-06 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008849+0000", 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47793379000000002 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008854+0000", 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0340000000000004e-06 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008927+0000", 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3088000000000002e-05 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009526+0000", 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059936699999999996 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.672 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=132, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.5708475209999992, 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44531548500000001, 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:57.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010331+0000", 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9600000000000001e-07 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010332+0000", 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.454e-06 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:57.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455554+0000", 2024-09-16T04:08:57.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44522236199999998 2024-09-16T04:08:57.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:57.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455561+0000", 2024-09-16T04:08:57.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4189999999999999e-06 2024-09-16T04:08:57.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455608+0000", 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6928000000000002e-05 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455646+0000", 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8126000000000001e-05 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:57.676 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:57.677 INFO:tasks.rados.rados.0.smithi016.stdout:322: expect (ObjNum 132 snap 27 seq_num 132) 2024-09-16T04:08:57.678 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:08:57.679 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:57.676+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:57.700 INFO:tasks.rados.rados.0.smithi016.stdout:323: finishing copy_from to smithi01639990-42 2024-09-16T04:08:57.700 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 42 v 22 (ObjNum 114 snap 19 seq_num 114) dirty exists 2024-09-16T04:08:57.700 INFO:tasks.rados.rados.0.smithi016.stdout:328: finishing write tid 1 to smithi01639990-25 2024-09-16T04:08:57.700 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 7 v 66 (ObjNum 120 snap 20 seq_num 120) dirty exists 2024-09-16T04:08:57.700 INFO:tasks.rados.rados.0.smithi016.stdout:322: done (8 left) 2024-09-16T04:08:57.700 INFO:tasks.rados.rados.0.smithi016.stdout:323: done (7 left) 2024-09-16T04:08:57.700 INFO:tasks.rados.rados.0.smithi016.stdout:325: done (6 left) 2024-09-16T04:08:57.700 INFO:tasks.rados.rados.0.smithi016.stdout:326: done (5 left) 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:333: snap_create 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:328: finishing write tid 2 to smithi01639990-25 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:328: finishing write tid 3 to smithi01639990-25 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:328: finishing write tid 4 to smithi01639990-25 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:328: finishing write tid 5 to smithi01639990-25 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:330: finishing write tid 1 to smithi01639990-23 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:330: finishing write tid 2 to smithi01639990-23 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:330: finishing write tid 3 to smithi01639990-23 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:330: finishing write tid 4 to smithi01639990-23 2024-09-16T04:08:57.701 INFO:tasks.rados.rados.0.smithi016.stdout:331: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T04:08:57.704 INFO:tasks.rados.rados.0.smithi016.stdout:329: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T04:08:57.722 INFO:tasks.rados.rados.0.smithi016.stdout:332: finishing copy_from racing read to smithi01639990-26 2024-09-16T04:08:57.722 INFO:tasks.rados.rados.0.smithi016.stdout:332: finishing copy_from to smithi01639990-26 2024-09-16T04:08:57.722 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 26 v 60 (ObjNum 118 snap 20 seq_num 118) dirty exists 2024-09-16T04:08:57.778 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:57.777+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:57.823 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:57.823 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:57.824 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:57.824 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:57.832 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:08:57.879 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:08:57.878+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:08:57.955 INFO:tasks.rados.rados.0.smithi016.stdout:328: finishing write tid 6 to smithi01639990-25 2024-09-16T04:08:57.955 INFO:tasks.rados.rados.0.smithi016.stdout:328: oid 25 updating version 0 to 45 2024-09-16T04:08:57.955 INFO:tasks.rados.rados.0.smithi016.stdout:328: oid 25 updating version 45 to 47 2024-09-16T04:08:57.955 INFO:tasks.rados.rados.0.smithi016.stdout:328: oid 25 updating version 47 to 48 2024-09-16T04:08:57.955 INFO:tasks.rados.rados.0.smithi016.stdout:328: oid 25 version 48 is already newer than 46 2024-09-16T04:08:57.955 INFO:tasks.rados.rados.0.smithi016.stdout:328: oid 25 updating version 48 to 49 2024-09-16T04:08:57.955 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 25 v 49 (ObjNum 136 snap 28 seq_num 136) dirty exists 2024-09-16T04:08:57.955 INFO:tasks.rados.rados.0.smithi016.stdout:328: left oid 25 (ObjNum 136 snap 28 seq_num 136) 2024-09-16T04:08:57.975 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:57.975 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:57.975 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:57.976 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:57.976 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:57.980 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:57.979+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:57.985 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:08:58.081 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:08:58.079+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:08:58.089 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:58.089 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:58.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:58.089 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 50.992847996999998, 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:58.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:08:58.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 39.657331026999998, 2024-09-16T04:08:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:08:58.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 39.197380799999998, 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.098 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.102 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.873868195, 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:08:58.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.616084336999997, 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078488229999999996, 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471072+0000", 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0279999999999999e-06 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471418+0000", 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034541800000000001 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471457+0000", 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471492+0000", 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471798+0000", 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030615599999999997 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471812+0000", 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.448e-05 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478881+0000", 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070685549999999998 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478920+0000", 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9072000000000001e-05 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.646969868999999, 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084259260000000002, 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.91e-07 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440211+0000", 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4978999999999999e-05 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440257+0000", 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6569000000000001e-05 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440268+0000", 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1056000000000001e-05 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440348+0000", 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9795000000000004e-05 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440364+0000", 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5996999999999999e-05 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448570+0000", 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082059590000000005 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448611+0000", 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.138e-05 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.bs7 57/56 [PushOp(3:d683cb60:::smithi01639990-49:head, version: 53'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d683cb60:::smithi01639990-49:head@53'38, size: 2310144, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.526387698000001, 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.043702234999999999, 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:08:58.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560769+0000", 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0049999999999999e-06 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.561301+0000", 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053276500000000004 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603955+0000", 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042653605999999997 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603967+0000", 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2065000000000001e-05 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604063+0000", 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5599000000000004e-05 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604470+0000", 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040719500000000001 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=252, reqid=osd.3.0:252, at_version=57'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.177661558, 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27497091299999998, 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909517+0000", 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2886000000000001e-05 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183847+0000", 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27432981400000001 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183853+0000", 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2929999999999997e-06 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183902+0000", 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9008999999999998e-05 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183911+0000", 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184435+0000", 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052493699999999995 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184465+0000", 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9292e-05 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=253, reqid=osd.3.0:253, at_version=57'67, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.177033526999999, 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50851210899999999, 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910123+0000", 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910136+0000", 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.331e-05 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416910+0000", 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50677369000000005 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416919+0000", 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8290000000000003e-06 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416975+0000", 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5983999999999998e-05 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416985+0000", 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418599+0000", 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001614515 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418634+0000", 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4588999999999997e-05 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 64/56 ECSubWrite(tid=285, reqid=client.4440.0:535, at_version=0'0, trim_to=0'0, roll_forward_to=64'72))", 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 18.545586457999999, 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012095003, 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541570+0000", 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.173e-06 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541582+0000", 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2106999999999999e-05 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552877+0000", 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011294692 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552889+0000", 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1875000000000001e-05 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552958+0000", 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9510000000000004e-05 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552967+0000", 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6240000000000008e-06 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553631+0000", 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066445299999999996 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553664+0000", 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2568999999999999e-05 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 68/46 ECSubWrite(tid=65, reqid=osd.4.0:65, at_version=68'22, trim_to=0'0, roll_forward_to=67'20))", 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.5458602, 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010787494999999999, 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541313+0000", 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7431e-05 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541369+0000", 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6311999999999999e-05 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541376+0000", 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5729999999999996e-06 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544486+0000", 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031104150000000001 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544495+0000", 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0960000000000007e-06 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552010+0000", 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075148200000000002 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552083+0000", 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2661000000000001e-05 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 68/56 ECSubWrite(tid=329, reqid=osd.3.0:329, at_version=68'68, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.544406479999999, 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093665009999999993, 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542750+0000", 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.23e-07 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542770+0000", 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0679999999999999e-05 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542842+0000", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1391999999999997e-05 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542850+0000", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2320000000000001e-06 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545061+0000", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022114220000000002 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545070+0000", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6780000000000006e-06 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552088+0000", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070176939999999997 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552115+0000", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7679999999999999e-05 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds2 69/56 ECSubWrite(tid=201, reqid=client.4440.0:654, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 13.546568616, 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090194439999999997, 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1400000000000001e-07 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540777+0000", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000189886 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540803+0000", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6466e-05 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540811+0000", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0730000000000005e-06 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540879+0000", 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8003999999999995e-05 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540889+0000", 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8460000000000003e-06 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.549571+0000", 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086822059999999996 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.549606+0000", 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4749000000000002e-05 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=336, reqid=osd.3.0:336, at_version=69'70, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 13.543462436, 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077691619999999996, 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.05e-07 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543708+0000", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4431e-05 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543897+0000", 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018918 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543906+0000", 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3349999999999996e-06 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543954+0000", 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7793999999999997e-05 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543963+0000", 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4259999999999992e-06 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551434+0000", 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0074706800000000004 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551462+0000", 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8110999999999999e-05 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=338, reqid=osd.3.0:338, at_version=69'72, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 13.542768252, 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010064081000000001, 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.151 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544388+0000", 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9800000000000003e-07 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544409+0000", 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0993e-05 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544442+0000", 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2937e-05 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544449+0000", 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.588e-06 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547347+0000", 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028976599999999998 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547357+0000", 2024-09-16T04:08:58.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6479999999999998e-06 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554419+0000", 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070620819999999999 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554451+0000", 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2475000000000003e-05 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s6 69/56 ECSubWrite(tid=202, reqid=osd.7.0:202, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 13.541431457, 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010752746000000001, 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:08:58.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545725+0000", 2024-09-16T04:08:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4199999999999995e-07 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545750+0000", 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4915000000000001e-05 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546880+0000", 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011307870000000001 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546891+0000", 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0825000000000001e-05 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551495+0000", 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004604131 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551504+0000", 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6230000000000003e-06 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556450+0000", 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049460210000000001 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556477+0000", 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6801999999999999e-05 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds2 69/56 ECSubWrite(tid=203, reqid=osd.7.0:203, at_version=69'28, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 13.541402421000001, 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078627160000000005, 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.91e-07 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545769+0000", 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5562000000000001e-05 2024-09-16T04:08:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546923+0000", 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001154457 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546936+0000", 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3183000000000001e-05 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547734+0000", 2024-09-16T04:08:58.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00079775500000000004 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547744+0000", 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5489999999999995e-06 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.553577+0000", 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058334809999999997 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.553616+0000", 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8438e-05 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 74/27 ECSubWrite(tid=161, reqid=osd.5.0:161, at_version=74'47, trim_to=0'0, roll_forward_to=73'45))", 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:49.456560+0000", 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 8.6305953570000007, 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0076668730000000003, 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.165 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456560+0000", 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456560+0000", 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456560+0000", 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8799999999999999e-07 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456580+0000", 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0119000000000002e-05 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456649+0000", 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8806000000000005e-05 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456656+0000", 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9240000000000001e-06 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456720+0000", 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4368000000000004e-05 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.456730+0000", 2024-09-16T04:08:58.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5759999999999995e-06 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.464193+0000", 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0074625689999999996 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.464227+0000", 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4323000000000003e-05 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 82/56 ECSubWrite(tid=197, reqid=client.4440.0:785, at_version=82'40, trim_to=0'0, roll_forward_to=74'38))", 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.59438882000000004, 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0095796900000000001, 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6400000000000001e-07 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492863+0000", 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6371000000000003e-05 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492883+0000", 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9507000000000001e-05 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492894+0000", 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0998999999999999e-05 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492958+0000", 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3898000000000004e-05 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492968+0000", 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0012e-05 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.502307+0000", 2024-09-16T04:08:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093395509999999998 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.502346+0000", 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9187999999999998e-05 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s5 82/56 ECSubWrite(tid=203, reqid=client.4440.0:808, at_version=82'31, trim_to=0'0, roll_forward_to=80'29))", 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.576184+0000", 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.51097115000000004, 2024-09-16T04:08:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0087388190000000001, 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.576184+0000", 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.576184+0000", 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.576185+0000", 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.06e-07 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.579980+0000", 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003795245 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.580013+0000", 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2743000000000001e-05 2024-09-16T04:08:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.580021+0000", 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8399999999999995e-06 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.580088+0000", 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7337999999999998e-05 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.580100+0000", 2024-09-16T04:08:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2335e-05 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.584876+0000", 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047755929999999999 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.584923+0000", 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7018999999999998e-05 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.179 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.179 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:58.180 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:08:58.182 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:58.180+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:803 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=64b] snapc 1d=[1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.72004663400000002, 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.72006972899999999, 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 803 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562173+0000", 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3200000000000005e-07 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562176+0000", 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9900000000000002e-06 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562183+0000", 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9480000000000002e-06 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562188+0000", 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1479999999999997e-06 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.565211+0000", 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030228519999999999 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.565509+0000", 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000297578 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.566353+0000", 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00084484499999999999 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.568092+0000", 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017385720000000001 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:804 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.72002308299999995, 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.72009316199999995, 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "waiting for rw locks", 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 804 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2e-07 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562199+0000", 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1119999999999999e-06 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562206+0000", 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9369999999999998e-06 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562211+0000", 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2569999999999998e-06 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.567444+0000", 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005233265 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.567504+0000", 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9769999999999999e-05 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: ], 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 2 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:58.294 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:58.281+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:58.294 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:08:58.334 INFO:tasks.rados.rados.0.smithi016.stdout:330: finishing write tid 5 to smithi01639990-23 2024-09-16T04:08:58.384 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:58.383+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:58.446 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:08:58.488 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:08:58.486+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:08:58.490 INFO:tasks.rados.rados.0.smithi016.stdout:328: done (5 left) 2024-09-16T04:08:58.490 INFO:tasks.rados.rados.0.smithi016.stdout:329: done (4 left) 2024-09-16T04:08:58.490 INFO:tasks.rados.rados.0.smithi016.stdout:331: done (3 left) 2024-09-16T04:08:58.491 INFO:tasks.rados.rados.0.smithi016.stdout:332: done (2 left) 2024-09-16T04:08:58.491 INFO:tasks.rados.rados.0.smithi016.stdout:333: done (1 left) 2024-09-16T04:08:58.491 INFO:tasks.rados.rados.0.smithi016.stdout:334: rmattr oid 28 current snap is 29 2024-09-16T04:08:58.491 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 33 current snap is 29 2024-09-16T04:08:58.492 INFO:tasks.rados.rados.0.smithi016.stdout:335: seq_num 138 ranges {0=770048,770048=540672,1310720=753664,2064384=262144} 2024-09-16T04:08:58.494 INFO:tasks.rados.rados.0.smithi016.stdout:335: writing smithi01639990-33 from 0 to 770048 tid 1 2024-09-16T04:08:58.498 INFO:tasks.rados.rados.0.smithi016.stdout:335: writing smithi01639990-33 from 770048 to 1310720 tid 2 2024-09-16T04:08:58.503 INFO:tasks.rados.rados.0.smithi016.stdout:335: writing smithi01639990-33 from 1310720 to 2064384 tid 3 2024-09-16T04:08:58.505 INFO:tasks.rados.rados.0.smithi016.stdout:335: writing smithi01639990-33 from 2064384 to 2326528 tid 4 2024-09-16T04:08:58.505 INFO:tasks.rados.rados.0.smithi016.stdout:336: delete oid 11 current snap is 29 2024-09-16T04:08:58.505 INFO:tasks.rados.rados.0.smithi016.stdout:335: finishing write tid 1 to smithi01639990-33 2024-09-16T04:08:58.505 INFO:tasks.rados.rados.0.smithi016.stdout:335: finishing write tid 2 to smithi01639990-33 2024-09-16T04:08:58.506 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 28 v 89 (ObjNum 109 snap 18 seq_num 109) dirty exists 2024-09-16T04:08:58.509 INFO:tasks.rados.rados.0.smithi016.stdout:335: finishing write tid 3 to smithi01639990-33 2024-09-16T04:08:58.510 INFO:tasks.rados.rados.0.smithi016.stdout:334: done (3 left) 2024-09-16T04:08:58.510 INFO:tasks.rados.rados.0.smithi016.stdout:336: done (2 left) 2024-09-16T04:08:58.510 INFO:tasks.rados.rados.0.smithi016.stdout:337: delete oid 3 current snap is 29 2024-09-16T04:08:58.512 INFO:tasks.rados.rados.0.smithi016.stdout:335: finishing write tid 4 to smithi01639990-33 2024-09-16T04:08:58.516 INFO:tasks.rados.rados.0.smithi016.stdout:335: finishing write tid 5 to smithi01639990-33 2024-09-16T04:08:58.517 INFO:tasks.rados.rados.0.smithi016.stdout:335: finishing write tid 6 to smithi01639990-33 2024-09-16T04:08:58.517 INFO:tasks.rados.rados.0.smithi016.stdout:335: oid 33 updating version 0 to 57 2024-09-16T04:08:58.517 INFO:tasks.rados.rados.0.smithi016.stdout:335: oid 33 updating version 57 to 59 2024-09-16T04:08:58.517 INFO:tasks.rados.rados.0.smithi016.stdout:335: oid 33 version 59 is already newer than 55 2024-09-16T04:08:58.517 INFO:tasks.rados.rados.0.smithi016.stdout:335: oid 33 version 59 is already newer than 56 2024-09-16T04:08:58.518 INFO:tasks.rados.rados.0.smithi016.stdout:335: oid 33 version 59 is already newer than 58 2024-09-16T04:08:58.518 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 33 v 59 (ObjNum 138 snap 29 seq_num 138) dirty exists 2024-09-16T04:08:58.518 INFO:tasks.rados.rados.0.smithi016.stdout:335: left oid 33 (ObjNum 138 snap 29 seq_num 138) 2024-09-16T04:08:58.527 INFO:tasks.rados.rados.0.smithi016.stdout:335: done (2 left) 2024-09-16T04:08:58.527 INFO:tasks.rados.rados.0.smithi016.stdout:337: done (1 left) 2024-09-16T04:08:58.527 INFO:tasks.rados.rados.0.smithi016.stdout:338: read oid 7 snap 19 2024-09-16T04:08:58.528 INFO:tasks.rados.rados.0.smithi016.stdout:338: expect deleted 2024-09-16T04:08:58.528 INFO:tasks.rados.rados.0.smithi016.stdout:339: snap_remove snap 28 2024-09-16T04:08:58.586 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:58.584+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:58.597 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:58.597 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:58.597 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:58.597 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:58.597 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.597 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 52/46 ECSubWrite(tid=210, reqid=client.4440.0:434, at_version=52'63, trim_to=0'0, roll_forward_to=50'60))", 2024-09-16T04:08:58.597 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 31.076935315, 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83621405000000004, 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.598 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8200000000000001e-07 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517790+0000", 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2540000000000001e-05 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.599 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352789+0000", 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83499984500000002 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352804+0000", 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4569999999999999e-05 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352928+0000", 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012396799999999999 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.600 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352943+0000", 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353943+0000", 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099977500000000006 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353981+0000", 2024-09-16T04:08:58.601 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7700999999999997e-05 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=217, reqid=client.4440.0:449, at_version=53'58, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.083425279, 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0096255799999999999, 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.602 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511277+0000", 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511278+0000", 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.068e-06 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.603 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.511623+0000", 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034493499999999999 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515041+0000", 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034178889999999999 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515048+0000", 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3479999999999998e-06 2024-09-16T04:08:58.604 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515115+0000", 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6469000000000003e-05 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.515126+0000", 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1187e-05 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520872+0000", 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0057460280000000002 2024-09-16T04:08:58.605 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.520902+0000", 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0655999999999999e-05 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=220, reqid=client.4440.0:452, at_version=53'61, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.074931761999999, 2024-09-16T04:08:58.606 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51827822899999998, 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.607 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519771+0000", 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.61e-07 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519788+0000", 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6699999999999999e-05 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036882+0000", 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51709414300000001 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.608 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036895+0000", 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3363999999999999e-05 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036964+0000", 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8734999999999998e-05 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036973+0000", 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0289999999999994e-06 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.609 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038014+0000", 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010410529999999999 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038049+0000", 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4443999999999998e-05 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.610 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:526 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~622592] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.964135374000001, 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01502261, 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 526 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.611 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630568+0000", 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5680000000000001e-06 2024-09-16T04:08:58.612 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631353+0000", 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078490700000000005 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631367+0000", 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3441000000000001e-05 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631375+0000", 2024-09-16T04:08:58.613 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8809999999999996e-06 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631465+0000", 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0673000000000004e-05 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631565+0000", 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9753000000000005e-05 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.614 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.632641+0000", 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001076097 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634418+0000", 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017769680000000001 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645537+0000", 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011119393999999999 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.615 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645589+0000", 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1928000000000002e-05 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:527 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.963298301000002, 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015053211, 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.616 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 527 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.617 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631405+0000", 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.08e-06 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631411+0000", 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9009999999999999e-06 2024-09-16T04:08:58.618 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631419+0000", 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631426+0000", 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4999999999999996e-06 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633558+0000", 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002131796 2024-09-16T04:08:58.619 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633663+0000", 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105833 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634106+0000", 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00044296899999999998 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.636380+0000", 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022740849999999999 2024-09-16T04:08:58.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646383+0000", 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01000238 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646457+0000", 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4258000000000006e-05 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.621 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:528 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.963223719999998, 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65958352099999995, 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 528 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.622 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631479+0000", 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9599999999999998e-07 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.623 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631483+0000", 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8160000000000004e-06 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631494+0000", 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0407999999999999e-05 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631501+0000", 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2269999999999996e-06 2024-09-16T04:08:58.624 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634655+0000", 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031544339999999998 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634732+0000", 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6495999999999994e-05 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646864+0000", 2024-09-16T04:08:58.625 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012131807999999999 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646965+0000", 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010173 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.291062+0000", 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64409670600000002 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.626 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s7 67/46 ECSubWrite(tid=62, reqid=osd.4.0:62, at_version=67'20, trim_to=0'0, roll_forward_to=66'18))", 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.061088377000001, 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090747759999999997, 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.627 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533614+0000", 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8199999999999999e-07 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.628 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533635+0000", 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1378000000000001e-05 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533659+0000", 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.374e-05 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.629 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533667+0000", 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2230000000000007e-06 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533785+0000", 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000117607 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.533795+0000", 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0180000000000001e-05 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542641+0000", 2024-09-16T04:08:58.630 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088456149999999994 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:42.542689+0000", 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7851e-05 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s2 68/56 ECSubWrite(tid=134, reqid=osd.0.0:134, at_version=68'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.052667400000001, 2024-09-16T04:08:58.631 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010826218, 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542035+0000", 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8599999999999998e-07 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542054+0000", 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8423999999999998e-05 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542081+0000", 2024-09-16T04:08:58.633 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7806000000000001e-05 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542089+0000", 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5789999999999998e-06 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542147+0000", 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.7978e-05 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542157+0000", 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9590000000000007e-06 2024-09-16T04:08:58.634 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552766+0000", 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010609143999999999 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552861+0000", 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4841999999999998e-05 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 68/56 ECSubWrite(tid=330, reqid=osd.3.0:330, at_version=68'75, trim_to=0'0, roll_forward_to=64'73))", 2024-09-16T04:08:58.635 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.051612444, 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094449849999999995, 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.636 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543090+0000", 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6000000000000003e-07 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543110+0000", 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9695e-05 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543139+0000", 2024-09-16T04:08:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8467999999999999e-05 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543148+0000", 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5349999999999993e-06 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543200+0000", 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1536999999999998e-05 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.638 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.543206+0000", 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9379999999999997e-06 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552495+0000", 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.009289538 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552535+0000", 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9613999999999997e-05 2024-09-16T04:08:58.639 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:695 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~589824] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.145675728000001, 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014512838, 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.640 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 695 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.641 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449028+0000", 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.917e-06 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450009+0000", 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000980367 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450023+0000", 2024-09-16T04:08:58.642 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.416e-05 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450032+0000", 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5820000000000008e-06 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450094+0000", 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1698000000000005e-05 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450210+0000", 2024-09-16T04:08:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116104 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451973+0000", 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001762958 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463392+0000", 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011419024999999999 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463450+0000", 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8251999999999997e-05 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463539+0000", 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8775000000000003e-05 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:696 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~753664] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:08:58.645 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.142543293999999, 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013764281, 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 696 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:08:58.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452161+0000", 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2579999999999998e-06 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453157+0000", 2024-09-16T04:08:58.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099543299999999999 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453172+0000", 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5917e-05 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453182+0000", 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3959999999999995e-06 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455292+0000", 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021103379999999998 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455406+0000", 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000114203 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456783+0000", 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001376377 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463544+0000", 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067608640000000001 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465825+0000", 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022816350000000002 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465923+0000", 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7860000000000002e-05 2024-09-16T04:08:58.650 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:697 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~475136] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.456443+0000", 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.138258666, 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012866471000000001, 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 697 2024-09-16T04:08:58.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456443+0000", 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456443+0000", 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456445+0000", 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6220000000000001e-06 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457074+0000", 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000629248 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457087+0000", 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2906e-05 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457096+0000", 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9449999999999999e-06 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.457956+0000", 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00086007 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458062+0000", 2024-09-16T04:08:58.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105559 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.459157+0000", 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00109566 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463617+0000", 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0044591639999999998 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469249+0000", 2024-09-16T04:08:58.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056324160000000003 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469310+0000", 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0881000000000002e-05 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:698 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~327680] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.458939+0000", 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.135763463, 2024-09-16T04:08:58.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010478334000000001, 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 698 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458939+0000", 2024-09-16T04:08:58.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458939+0000", 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.458940+0000", 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5770000000000001e-06 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460359+0000", 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001418334 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460372+0000", 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3025e-05 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460379+0000", 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4680000000000003e-06 2024-09-16T04:08:58.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460452+0000", 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.339e-05 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.460661+0000", 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00020814299999999999 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.461550+0000", 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00088961800000000001 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465933+0000", 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0043829330000000003 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469379+0000", 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0034461069999999999 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.469417+0000", 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7738999999999999e-05 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:699 3.11s0 3:8bfd968c:::smithi01639990-27:head [setxattr _header (57) in=7b] snapc 18=[18,16,15,13] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.471289+0000", 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.123413005, 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012440791, 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:58.662 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 699 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471289+0000", 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471289+0000", 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471290+0000", 2024-09-16T04:08:58.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1800000000000005e-07 2024-09-16T04:08:58.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471292+0000", 2024-09-16T04:08:58.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6180000000000002e-06 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471297+0000", 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7419999999999997e-06 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471302+0000", 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9320000000000004e-06 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471382+0000", 2024-09-16T04:08:58.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9864000000000006e-05 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471444+0000", 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2249999999999995e-05 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471785+0000", 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000340267 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472441+0000", 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00065623500000000002 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.483660+0000", 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011219632 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.483730+0000", 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9533e-05 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.667 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:700 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.123392786, 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014504263, 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.668 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 700 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471310+0000", 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.08e-07 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471312+0000", 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0540000000000002e-06 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471318+0000", 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5080000000000002e-06 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471322+0000", 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3629999999999998e-06 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472045+0000", 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00072327100000000005 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472133+0000", 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8528000000000003e-05 2024-09-16T04:08:58.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484130+0000", 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011996163000000001 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484213+0000", 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3800000000000004e-05 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.485814+0000", 2024-09-16T04:08:58.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016001679999999999 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.064067891000001, 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095992542, 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530636+0000", 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.522e-06 2024-09-16T04:08:58.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530651+0000", 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4980999999999999e-05 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625860+0000", 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095209590999999996 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625873+0000", 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2413e-05 2024-09-16T04:08:58.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625978+0000", 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010492099999999999 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625994+0000", 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6058000000000001e-05 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626583+0000", 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058931700000000001 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626627+0000", 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3739000000000003e-05 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:800 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~786432] snapc 1d=[1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:08:58.677 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.557628+0000", 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.037074246, 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0091165259999999998, 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 800 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.557628+0000", 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.557628+0000", 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.557629+0000", 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.322e-06 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.558420+0000", 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00079082200000000005 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.558430+0000", 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0291e-05 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.558435+0000", 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.933e-06 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.558480+0000", 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4987000000000001e-05 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.558584+0000", 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000103874 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.559596+0000", 2024-09-16T04:08:58.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001011542 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.560667+0000", 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010710380000000001 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.566665+0000", 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059980959999999996 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.566744+0000", 2024-09-16T04:08:58.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9621000000000002e-05 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:803 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=64b] snapc 1d=[1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0325297120000001, 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.77068146599999998, 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 803 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:08:58.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562173+0000", 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3200000000000005e-07 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562176+0000", 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9900000000000002e-06 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562183+0000", 2024-09-16T04:08:58.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9480000000000002e-06 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562188+0000", 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1479999999999997e-06 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.565211+0000", 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030228519999999999 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:58.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.565509+0000", 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000297578 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.566353+0000", 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00084484499999999999 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.568092+0000", 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017385720000000001 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.332782+0000", 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.76468974099999998 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.332854+0000", 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.216e-05 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.688 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 82/56 ECSubWriteReply(tid=41, last_complete=82'37, committed=1, applied=1))", 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.572277+0000", 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.0224252540000001, 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.754736132, 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572277+0000", 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572277+0000", 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572278+0000", 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5199999999999996e-07 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572281+0000", 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7290000000000002e-06 2024-09-16T04:08:58.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.326881+0000", 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.75459961200000003 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.326897+0000", 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6180000000000001e-05 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.326988+0000", 2024-09-16T04:08:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0477999999999997e-05 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.327013+0000", 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5480999999999999e-05 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s2 82/56 ECSubWrite(tid=203, reqid=client.4440.0:808, at_version=82'31, trim_to=0'0, roll_forward_to=80'29))", 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.576344+0000", 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.018357972, 2024-09-16T04:08:58.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01154703, 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.576344+0000", 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.576344+0000", 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.576345+0000", 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1399999999999997e-07 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.581679+0000", 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005334119 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.581710+0000", 2024-09-16T04:08:58.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0725999999999997e-05 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.581717+0000", 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8110000000000003e-06 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.581767+0000", 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9499000000000002e-05 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:58.695 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.581775+0000", 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6090000000000003e-06 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.587861+0000", 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060860719999999997 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.587891+0000", 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0580000000000002e-05 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.696 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.697 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:58.697 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:58.697 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:58.698 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:08:58.699 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:08:58.696+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:08:58.799 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:58.797+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:58.801 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:58.801 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:08:58.801 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:58.810 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:08:58.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:08:58.934 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:58.932+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:08:58.953 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:58.953 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:08:58.953 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:08:58.953 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:08:58.953 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:58.962 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:08:59.002 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:08:59.000+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:08:59.034 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:08:59.034 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:08:59.056 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:08:59.057 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:08:59.066 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:08:59.066 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:08:59.066 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:08:59.066 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:08:59.066 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.066 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:124 1.0 e22/19 1:30306672:devicehealth::main.db.0000000000000000:head v 22'86, mlcod=21'84)", 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 66.633155666999997, 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020095396000000001, 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.067 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430782+0000", 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9699999999999996e-07 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430789+0000", 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4660000000000002e-06 2024-09-16T04:08:59.068 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450028+0000", 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019238816999999998 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450042+0000", 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4487e-05 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.069 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450169+0000", 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126787 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450188+0000", 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450816+0000", 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062741399999999997 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.070 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450877+0000", 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0782999999999997e-05 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:197 3.es0 3:7f241f48:::smithi01639990-30:head [copy-from ver 23] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 50.853946319000002, 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.023242381999999999, 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.071 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 197 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.072 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6499999999999999e-07 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.073 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209995+0000", 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.388e-06 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210002+0000", 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5209999999999997e-06 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.074 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210008+0000", 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8780000000000003e-06 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8421999999999999e-05 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210099+0000", 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2469999999999997e-05 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.075 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227450+0000", 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017351235 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232352+0000", 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049017239999999997 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:59.076 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233168+0000", 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081616600000000001 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233233+0000", 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5012999999999996e-05 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:198 3.es0 3:7f241f48:::smithi01639990-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 50.853916009999999, 2024-09-16T04:08:59.077 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019673758, 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 198 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.078 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3099999999999998e-07 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210024+0000", 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2620000000000001e-06 2024-09-16T04:08:59.079 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210030+0000", 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9020000000000002e-06 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210037+0000", 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1480000000000003e-06 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.080 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210227+0000", 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00019055499999999999 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210269+0000", 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2246999999999997e-05 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229480+0000", 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019210896000000002 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.081 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229587+0000", 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107026 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229695+0000", 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107291 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:213 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:59.082 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 49.947837505000003, 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53275775299999995, 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 213 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.083 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116100+0000", 2024-09-16T04:08:59.084 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116103+0000", 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.174e-06 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116109+0000", 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4330000000000003e-06 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116114+0000", 2024-09-16T04:08:59.085 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1399999999999999e-06 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119298+0000", 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031846729999999998 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119365+0000", 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6484999999999998e-05 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:59.086 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119675+0000", 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00031021999999999999 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122365+0000", 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002689851 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648785+0000", 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52641971300000001 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.087 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648857+0000", 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2408000000000002e-05 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:214 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 49.947814829000002, 2024-09-16T04:08:59.088 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53416417900000002, 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 214 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2100000000000002e-07 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116125+0000", 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2649999999999999e-06 2024-09-16T04:08:59.090 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116132+0000", 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5190000000000003e-06 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116137+0000", 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9869999999999999e-06 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120036+0000", 2024-09-16T04:08:59.091 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038984979999999998 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120118+0000", 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2312999999999996e-05 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649191+0000", 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52907320199999996 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.092 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649279+0000", 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7585999999999998e-05 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.650286+0000", 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010073879999999999 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "replica_scrub(pg: 3.4s1,from:0'0,to:37'17,epoch:47/46,start:3:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:59.093 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 36.206013292000002, 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16799329900000001, 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9400000000000005e-07 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857929+0000", 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4719999999999997e-06 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025793+0000", 2024-09-16T04:08:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16786378299999999 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025803+0000", 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9629999999999993e-06 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025875+0000", 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2914e-05 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.096 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025887+0000", 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1345999999999999e-05 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025917+0000", 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0127000000000001e-05 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.097 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.4s1,RELEASE,e:47,from:1(0),reservation_nonce:0)", 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 36.036868317, 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26135687200000002, 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.098 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2800000000000005e-07 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027075+0000", 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3859999999999998e-06 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288302+0000", 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26122710999999998 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288308+0000", 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6189999999999999e-06 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288392+0000", 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4549000000000007e-05 2024-09-16T04:08:59.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288414+0000", 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1576000000000001e-05 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288425+0000", 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1704e-05 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:385 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.597866045000004, 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01911771, 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 385 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466071+0000", 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466073+0000", 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4389999999999999e-06 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466084+0000", 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1157e-05 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466101+0000", 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6334999999999999e-05 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466111+0000", 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466255+0000", 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014411 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466263+0000", 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.858e-06 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474564+0000", 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0083014019999999994 2024-09-16T04:08:59.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.474734+0000", 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00016996599999999999 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.475786+0000", 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010515909999999999 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.482124+0000", 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063382530000000003 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485117+0000", 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0029928120000000001 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485189+0000", 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1868999999999999e-05 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.107 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:386 3.8s0 3:17b72cd6:::smithi01639990-47:head [rollback a] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.597794311999998, 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019422101000000001, 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.108 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 386 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466143+0000", 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8100000000000002e-07 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466147+0000", 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6559999999999998e-06 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466155+0000", 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6240000000000002e-06 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466162+0000", 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4850000000000003e-06 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466343+0000", 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000181077 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466349+0000", 2024-09-16T04:08:59.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1229999999999997e-06 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476407+0000", 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010057885000000001 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.476516+0000", 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000108738 2024-09-16T04:08:59.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.477098+0000", 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000582024 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484050+0000", 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0069524950000000004 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485519+0000", 2024-09-16T04:08:59.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014691280000000001 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.485565+0000", 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5185000000000002e-05 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:387 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.597766211, 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51761721800000005, 2024-09-16T04:08:59.114 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 387 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:59.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6399999999999998e-07 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466173+0000", 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.396e-06 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466179+0000", 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9290000000000003e-06 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466184+0000", 2024-09-16T04:08:59.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9760000000000003e-06 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466397+0000", 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021269400000000001 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466406+0000", 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0769999999999997e-06 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479302+0000", 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012896014000000001 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479415+0000", 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112767 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479914+0000", 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00049861199999999995 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484063+0000", 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004149158 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983716+0000", 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49965376500000003 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983788+0000", 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1465999999999996e-05 2024-09-16T04:08:59.121 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/46 ECSubWriteReply(tid=59, last_complete=48'38, committed=1, applied=1))", 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.579725199000002, 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50006588100000005, 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2499999999999999e-07 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484214+0000", 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6440000000000001e-06 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983573+0000", 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49935939200000001 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983581+0000", 2024-09-16T04:08:59.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983628+0000", 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7642999999999998e-05 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.984278+0000", 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00064939900000000005 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "rep_scrubmap(3.8s0 e46 from shard 10(6))", 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.077623907000003, 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21666559099999999, 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986314+0000", 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5600000000000005e-07 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986319+0000", 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0950000000000003e-06 2024-09-16T04:08:59.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202862+0000", 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21654300900000001 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202872+0000", 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0077e-05 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202931+0000", 2024-09-16T04:08:59.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9568000000000003e-05 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202942+0000", 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0139999999999999e-05 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202979+0000", 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6946000000000002e-05 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s3 53/48 ECSubWrite(tid=39, reqid=osd.4.0:39, at_version=53'24, trim_to=0'0, roll_forward_to=51'22))", 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.574999119000001, 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017845360000000001, 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488938+0000", 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2000000000000001e-07 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.488964+0000", 2024-09-16T04:08:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6418e-05 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502624+0000", 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013659439000000001 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502636+0000", 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2401999999999999e-05 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502702+0000", 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6006999999999993e-05 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.502712+0000", 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8870000000000004e-06 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506740+0000", 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0040279319999999997 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.133 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506783+0000", 2024-09-16T04:08:59.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3055e-05 2024-09-16T04:08:59.134 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.134 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.134 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.134 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:446 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:59.134 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:59.134 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.560397460000001, 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67862900900000001, 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 446 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503541+0000", 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3680000000000001e-06 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503545+0000", 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.561e-06 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503553+0000", 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8939999999999993e-06 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503559+0000", 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.224e-06 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505239+0000", 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001679739 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505303+0000", 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4219000000000003e-05 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505673+0000", 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000369374 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.507082+0000", 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014090330000000001 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182055+0000", 2024-09-16T04:08:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67497357700000005 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182168+0000", 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011302 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:447 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.560367954, 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.680085987, 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 447 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503570+0000", 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.44e-07 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503576+0000", 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4559999999999997e-06 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503602+0000", 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6264000000000001e-05 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503610+0000", 2024-09-16T04:08:59.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4109999999999999e-06 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505945+0000", 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023353390000000001 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506004+0000", 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9410999999999998e-05 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182603+0000", 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67659811199999997 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182695+0000", 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1951000000000001e-05 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.183655+0000", 2024-09-16T04:08:59.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00096039900000000002 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.555295629, 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.299096274, 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:59.146 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508642+0000", 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5400000000000002e-07 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508645+0000", 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9249999999999999e-06 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807520+0000", 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.29887539899999999 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807534+0000", 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3643e-05 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807681+0000", 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014696699999999999 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807738+0000", 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6885999999999997e-05 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.554914163999999, 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67354323000000005, 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9500000000000002e-07 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509027+0000", 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4429999999999999e-06 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181881+0000", 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67285377899999999 2024-09-16T04:08:59.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181891+0000", 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0479000000000001e-05 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181961+0000", 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0370000000000006e-05 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182566+0000", 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060456400000000003 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.4s1 62/56 ECSubRead(tid=8, to_read={3:26eb5942:::smithi01639990-23:head=0,4096,0}, subchunks={3:26eb5942:::smithi01639990-23:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 21.416283333999999, 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64312231099999995, 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:08:59.154 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2500000000000004e-07 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647678+0000", 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3421000000000002e-05 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290437+0000", 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64275884900000002 2024-09-16T04:08:59.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290450+0000", 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2858e-05 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290545+0000", 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5463000000000004e-05 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290776+0000", 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000230895 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:705 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.571196884999999, 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.037755205, 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.158 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 705 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492741+0000", 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6199999999999997e-07 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492743+0000", 2024-09-16T04:08:59.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7530000000000002e-06 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492750+0000", 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7710000000000001e-06 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492756+0000", 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.524e-06 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494038+0000", 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012823310000000001 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494140+0000", 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010216400000000001 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494664+0000", 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052402199999999996 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:08:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495429+0000", 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00076470399999999995 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530435+0000", 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035005738000000002 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530495+0000", 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0535999999999998e-05 2024-09-16T04:08:59.163 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:706 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.571171161000001, 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.039505024, 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:08:59.164 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 706 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:59.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0600000000000001e-07 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492769+0000", 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7420000000000002e-06 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:08:59.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492776+0000", 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0480000000000001e-06 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492781+0000", 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1499999999999998e-06 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495503+0000", 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027220199999999999 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:08:59.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495628+0000", 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000125041 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530799+0000", 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035171257999999997 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530897+0000", 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7608999999999993e-05 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:08:59.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:08:59.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:08:59.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.532271+0000", 2024-09-16T04:08:59.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00137375 2024-09-16T04:08:59.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.169 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:08:59.169 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:08:59.169 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:08:59.170 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:08:59.171 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:08:59.171 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:08:59.172 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:08:59.172 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:08:59.173 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:08:59.173 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:08:59.173 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:08:59.174 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:08:59.174 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:08:59.174 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:08:59.174 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:59.169+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:08:59.185 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:08:59.185 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:08:59.207 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:08:59.207 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:08:59.228 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:08:59.228 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:08:59.250 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:08:59.250 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:08:59.271 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:59.270+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:59.272 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:08:59.272 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:08:59.283 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:59.283 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:59.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:08:59.372 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:59.371+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:59.438 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:59.439 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:59.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:08:59.491 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:08:59.489+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:08:59.492 INFO:tasks.rados.rados.0.smithi016.stdout:339: done (2 left) 2024-09-16T04:08:59.492 INFO:tasks.rados.rados.0.smithi016.stdout:340: rmattr oid 5 current snap is 29 2024-09-16T04:08:59.492 INFO:tasks.rados.rados.0.smithi016.stdout:330: finishing write tid 6 to smithi01639990-23 2024-09-16T04:08:59.492 INFO:tasks.rados.rados.0.smithi016.stdout:330: oid 23 updating version 0 to 33 2024-09-16T04:08:59.492 INFO:tasks.rados.rados.0.smithi016.stdout:330: oid 23 updating version 33 to 37 2024-09-16T04:08:59.492 INFO:tasks.rados.rados.0.smithi016.stdout:330: oid 23 version 37 is already newer than 34 2024-09-16T04:08:59.492 INFO:tasks.rados.rados.0.smithi016.stdout:330: oid 23 version 37 is already newer than 36 2024-09-16T04:08:59.492 INFO:tasks.rados.rados.0.smithi016.stdout:330: oid 23 version 37 is already newer than 35 2024-09-16T04:08:59.493 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 23 v 37 (ObjNum 137 snap 28 seq_num 137) dirty exists 2024-09-16T04:08:59.493 INFO:tasks.rados.rados.0.smithi016.stdout:330: left oid 23 (ObjNum 137 snap 28 seq_num 137) 2024-09-16T04:08:59.493 INFO:tasks.rados.rados.0.smithi016.stdout:330: done (2 left) 2024-09-16T04:08:59.493 INFO:tasks.rados.rados.0.smithi016.stdout:338: done (1 left) 2024-09-16T04:08:59.493 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 1 current snap is 29 2024-09-16T04:08:59.493 INFO:tasks.rados.rados.0.smithi016.stdout:341: seq_num 141 ranges {1982464=425984,2408448=573440,2981888=425984,3407872=524288,3932160=425984} 2024-09-16T04:08:59.505 INFO:tasks.rados.rados.0.smithi016.stdout:341: writing smithi01639990-1 from 1982464 to 2408448 tid 1 2024-09-16T04:08:59.509 INFO:tasks.rados.rados.0.smithi016.stdout:341: writing smithi01639990-1 from 2408448 to 2981888 tid 2 2024-09-16T04:08:59.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:08:59.513 INFO:tasks.rados.rados.0.smithi016.stdout:341: writing smithi01639990-1 from 2981888 to 3407872 tid 3 2024-09-16T04:08:59.517 INFO:tasks.rados.rados.0.smithi016.stdout:341: writing smithi01639990-1 from 3407872 to 3932160 tid 4 2024-09-16T04:08:59.520 INFO:tasks.rados.rados.0.smithi016.stdout:341: writing smithi01639990-1 from 3932160 to 4358144 tid 5 2024-09-16T04:08:59.520 INFO:tasks.rados.rados.0.smithi016.stdout:342: delete oid 26 current snap is 29 2024-09-16T04:08:59.521 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 5 v 38 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T04:08:59.521 INFO:tasks.rados.rados.0.smithi016.stdout:341: finishing write tid 1 to smithi01639990-1 2024-09-16T04:08:59.527 INFO:tasks.rados.rados.0.smithi016.stdout:340: done (2 left) 2024-09-16T04:08:59.527 INFO:tasks.rados.rados.0.smithi016.stdout:342: done (1 left) 2024-09-16T04:08:59.528 INFO:tasks.rados.rados.0.smithi016.stdout:343: snap_create 2024-09-16T04:08:59.530 INFO:tasks.rados.rados.0.smithi016.stdout:341: finishing write tid 2 to smithi01639990-1 2024-09-16T04:08:59.531 INFO:tasks.rados.rados.0.smithi016.stdout:341: finishing write tid 3 to smithi01639990-1 2024-09-16T04:08:59.532 INFO:tasks.rados.rados.0.smithi016.stdout:341: finishing write tid 4 to smithi01639990-1 2024-09-16T04:08:59.532 INFO:tasks.rados.rados.0.smithi016.stdout:341: finishing write tid 5 to smithi01639990-1 2024-09-16T04:08:59.533 INFO:tasks.rados.rados.0.smithi016.stdout:341: finishing write tid 6 to smithi01639990-1 2024-09-16T04:08:59.534 INFO:tasks.rados.rados.0.smithi016.stdout:341: finishing write tid 7 to smithi01639990-1 2024-09-16T04:08:59.534 INFO:tasks.rados.rados.0.smithi016.stdout:341: oid 1 updating version 0 to 78 2024-09-16T04:08:59.534 INFO:tasks.rados.rados.0.smithi016.stdout:341: oid 1 updating version 78 to 79 2024-09-16T04:08:59.534 INFO:tasks.rados.rados.0.smithi016.stdout:341: oid 1 version 79 is already newer than 76 2024-09-16T04:08:59.535 INFO:tasks.rados.rados.0.smithi016.stdout:341: oid 1 version 79 is already newer than 77 2024-09-16T04:08:59.535 INFO:tasks.rados.rados.0.smithi016.stdout:341: oid 1 updating version 79 to 80 2024-09-16T04:08:59.535 INFO:tasks.rados.rados.0.smithi016.stdout:341: oid 1 version 80 is already newer than 75 2024-09-16T04:08:59.535 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 1 v 80 (ObjNum 141 snap 29 seq_num 141) dirty exists 2024-09-16T04:08:59.535 INFO:tasks.rados.rados.0.smithi016.stdout:341: left oid 1 (ObjNum 141 snap 29 seq_num 141) 2024-09-16T04:08:59.574 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:59.572+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:59.576 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:59.576 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:59.585 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:08:59.675 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:08:59.674+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:08:59.714 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-09-16T04:08:59.714 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:08:59.714 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T04:08:59.714 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-09-16T04:08:59.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T04:08:59.737 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:59.737 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:59.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:08:59.776 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:08:59.774+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:08:59.877 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:08:59.875+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:08:59.899 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:08:59.899 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:08:59.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:08:59.977 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:08:59.976+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:00.056 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:00.056 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:00.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:09:00.078 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:00.077+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:00.172 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:00.172 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:00.179 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:09:00.178+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:09:00.180 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:09:00.280 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:00.278+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:00.286 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:00.286 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:00.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:09:00.381 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:09:00.379+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:09:00.401 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:00.402 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:00.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:09:00.435 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-16T04:09:00.436 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T04:09:00.441 INFO:tasks.rados.rados.0.smithi016.stdout:341: done (1 left) 2024-09-16T04:09:00.441 INFO:tasks.rados.rados.0.smithi016.stdout:343: done (0 left) 2024-09-16T04:09:00.441 INFO:tasks.rados.rados.0.smithi016.stdout:344: setattr oid 1 current snap is 30 2024-09-16T04:09:00.441 INFO:tasks.rados.rados.0.smithi016.stdout:345: delete oid 2 current snap is 30 2024-09-16T04:09:00.447 INFO:tasks.rados.rados.0.smithi016.stdout:345: done (1 left) 2024-09-16T04:09:00.447 INFO:tasks.rados.rados.0.smithi016.stdout:346: snap_create 2024-09-16T04:09:00.450 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 1 v 81 (ObjNum 141 snap 29 seq_num 141) dirty exists 2024-09-16T04:09:00.481 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:00.479+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:00.526 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:00.526 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:00.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:09:00.582 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:09:00.580+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:09:00.644 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:00.644 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:00.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:09:00.692 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:00.690+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:00.784 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:09:00.782+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:09:00.798 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:00.798 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:00.808 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:09:00.886 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:09:00.884+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:09:00.923 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:00.923 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:00.933 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:09:00.986 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:09:00.985+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:09:01.036 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:01.117 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:01.119 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:09:01.119 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:01.117+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:01.220 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:01.218+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:01.263 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:01.263 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:01.272 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:09:01.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:09:01.320 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:09:01.319+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:09:01.375 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:01.375 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:01.383 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:09:01.421 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:01.420+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:01.428 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:09:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:09:01.439 INFO:tasks.rados.rados.0.smithi016.stdout:344: done (1 left) 2024-09-16T04:09:01.439 INFO:tasks.rados.rados.0.smithi016.stdout:346: done (0 left) 2024-09-16T04:09:01.439 INFO:tasks.rados.rados.0.smithi016.stdout:347: read oid 9 snap -1 2024-09-16T04:09:01.439 INFO:tasks.rados.rados.0.smithi016.stdout:347: expect (ObjNum 118 snap 20 seq_num 118) 2024-09-16T04:09:01.439 INFO:tasks.rados.rados.0.smithi016.stdout:348: copy_from oid 38 from oid 27 current snap is 31 2024-09-16T04:09:01.439 INFO:tasks.rados.rados.0.smithi016.stdout:349: delete oid 12 current snap is 31 2024-09-16T04:09:01.448 INFO:tasks.rados.rados.0.smithi016.stdout:349: done (2 left) 2024-09-16T04:09:01.448 INFO:tasks.rados.rados.0.smithi016.stdout:350: delete oid 34 current snap is 31 2024-09-16T04:09:01.453 INFO:tasks.rados.rados.0.smithi016.stdout:347: expect (ObjNum 118 snap 20 seq_num 118) 2024-09-16T04:09:01.456 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:09:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:09:01.477 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:09:01.477 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:09:01.497 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:09:01.497 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:09:01.519 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:09:01.519 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:09:01.522 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:09:01.520+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:09:01.534 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:01.535 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:01.541 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:09:01.541 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:09:01.544 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:09:01.561 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:09:01.562 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:09:01.583 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:09:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:09:01.587 INFO:tasks.rados.rados.0.smithi016.stdout:348: finishing copy_from racing read to smithi01639990-38 2024-09-16T04:09:01.588 INFO:tasks.rados.rados.0.smithi016.stdout:348: finishing copy_from to smithi01639990-38 2024-09-16T04:09:01.588 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 38 v 28 (ObjNum 127 snap 23 seq_num 127) dirty exists 2024-09-16T04:09:01.588 INFO:tasks.rados.rados.0.smithi016.stdout:347: done (2 left) 2024-09-16T04:09:01.588 INFO:tasks.rados.rados.0.smithi016.stdout:348: done (1 left) 2024-09-16T04:09:01.588 INFO:tasks.rados.rados.0.smithi016.stdout:350: done (0 left) 2024-09-16T04:09:01.588 INFO:tasks.rados.rados.0.smithi016.stdout:351: read oid 28 snap -1 2024-09-16T04:09:01.588 INFO:tasks.rados.rados.0.smithi016.stdout:351: expect (ObjNum 109 snap 18 seq_num 109) 2024-09-16T04:09:01.588 INFO:tasks.rados.rados.0.smithi016.stdout:352: snap_remove snap 30 2024-09-16T04:09:01.607 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:09:01.608 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:09:01.623 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:01.621+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:01.629 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:09:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:09:01.649 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:01.649 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:01.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:01.649 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:01.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.649 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:09:01.649 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 52.543639689999999, 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013925085, 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103016+0000", 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5499999999999999e-07 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103038+0000", 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2416e-05 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103157+0000", 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000118439 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103166+0000", 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9819999999999997e-06 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103225+0000", 2024-09-16T04:09:01.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8996000000000002e-05 2024-09-16T04:09:01.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103235+0000", 2024-09-16T04:09:01.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9909999999999997e-06 2024-09-16T04:09:01.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116899+0000", 2024-09-16T04:09:01.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013664549999999999 2024-09-16T04:09:01.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116941+0000", 2024-09-16T04:09:01.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1455999999999997e-05 2024-09-16T04:09:01.656 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 52.543614388999998, 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013929731000000001, 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5899999999999998e-07 2024-09-16T04:09:01.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103057+0000", 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5778e-05 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103193+0000", 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0001359 2024-09-16T04:09:01.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103199+0000", 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1279999999999996e-06 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107717+0000", 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045183769999999996 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107726+0000", 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5420000000000005e-06 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116950+0000", 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092240229999999996 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116970+0000", 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0724e-05 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.661 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 42/27 ECSubWrite(tid=156, reqid=client.4440.0:345, at_version=42'30, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.202183173000002, 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011544518, 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444472+0000", 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.444473+0000", 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4499999999999996e-07 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447203+0000", 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.002729834 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447277+0000", 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4078999999999993e-05 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447283+0000", 2024-09-16T04:09:01.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4169999999999997e-06 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447458+0000", 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00017450300000000001 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.447469+0000", 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1062e-05 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.455978+0000", 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085090730000000007 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.456017+0000", 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8804999999999999e-05 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/27 ECSubWriteReply(tid=88, last_complete=42'39, committed=1, applied=1))", 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:09:01.666 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.198209155999997, 2024-09-16T04:09:01.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49630058100000002, 2024-09-16T04:09:01.667 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.667 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.667 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:01.667 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:09:01.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448447+0000", 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448451+0000", 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3030000000000001e-06 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944646+0000", 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49619510999999999 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944655+0000", 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8480000000000007e-06 2024-09-16T04:09:01.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944717+0000", 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2065999999999998e-05 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944747+0000", 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9505999999999998e-05 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDScrubReserve(3.0s7,QREQUEST,e:47,from:11(0),reservation_nonce:4)", 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.619487341999999, 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17961150300000001, 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:09:01.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027169+0000", 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027173+0000", 2024-09-16T04:09:01.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1300000000000003e-06 2024-09-16T04:09:01.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206636+0000", 2024-09-16T04:09:01.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17946284700000001 2024-09-16T04:09:01.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206644+0000", 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9650000000000008e-06 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206712+0000", 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8215999999999999e-05 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206735+0000", 2024-09-16T04:09:01.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2789000000000002e-05 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206779+0000", 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4669000000000002e-05 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:414 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~770048] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:09:01.675 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.162113044999998, 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012099706, 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 414 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484544+0000", 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485808+0000", 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012637099999999999 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485825+0000", 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7722000000000001e-05 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485834+0000", 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9749999999999996e-06 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485905+0000", 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1131999999999996e-05 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.486012+0000", 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000106214 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487551+0000", 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015389830000000001 2024-09-16T04:09:01.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.489803+0000", 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0022524839999999999 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496554+0000", 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067513879999999997 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496642+0000", 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7385000000000004e-05 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:416 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~294912 in=294912b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.156032633999999, 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16418209, 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:01.681 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 416 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:09:01.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490624+0000", 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3069999999999999e-06 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490921+0000", 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00029678400000000002 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490932+0000", 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1239e-05 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490938+0000", 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0379999999999996e-06 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490980+0000", 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2401999999999998e-05 2024-09-16T04:09:01.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491037+0000", 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6768000000000003e-05 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491591+0000", 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055345000000000004 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492749+0000", 2024-09-16T04:09:01.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0011580970000000001 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654745+0000", 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16199656800000001 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654805+0000", 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9437e-05 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:417 3.cs0 3:38c68ed3:::smithi01639990-42:head [setxattr _header (57) in=64b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.155702615000003, 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67294876800000003, 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 417 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.687 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2699999999999999e-07 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490956+0000", 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0130000000000002e-06 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490961+0000", 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1510000000000003e-06 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490966+0000", 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7960000000000003e-06 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491934+0000", 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00096816399999999998 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492020+0000", 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5827000000000003e-05 2024-09-16T04:09:01.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492479+0000", 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045912299999999997 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492973+0000", 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049418400000000005 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:01.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163802+0000", 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67082863599999998 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163901+0000", 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9246999999999994e-05 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:418 3.cs0 3:38c68ed3:::smithi01639990-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:01.692 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.155681436000002, 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67426341400000001, 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 418 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8000000000000001e-07 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490976+0000", 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.973e-06 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490983+0000", 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4699999999999999e-06 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490988+0000", 2024-09-16T04:09:01.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5509999999999999e-06 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493081+0000", 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020931029999999998 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493165+0000", 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3988999999999998e-05 2024-09-16T04:09:01.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164330+0000", 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67116498999999996 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164423+0000", 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000004e-05 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.165237+0000", 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081446800000000005 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 52/27 ECSubWriteReply(tid=99, last_complete=52'26, committed=1, applied=1))", 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.153739745000003, 2024-09-16T04:09:01.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16189895200000001, 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492916+0000", 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1699999999999998e-07 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492919+0000", 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6970000000000002e-06 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654619+0000", 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16170007 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654628+0000", 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9080000000000001e-06 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654685+0000", 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7349999999999998e-05 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654814+0000", 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012941 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 52/27 ECSubWriteReply(tid=100, last_complete=52'27, committed=1, applied=1))", 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.153297672999997, 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67093597800000004, 2024-09-16T04:09:01.702 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493358+0000", 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493360+0000", 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2019999999999998e-06 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163618+0000", 2024-09-16T04:09:01.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67025769000000002 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163644+0000", 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6089999999999999e-05 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163713+0000", 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9326999999999995e-05 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164293+0000", 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00058030099999999997 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 63/56 ECSubWrite(tid=277, reqid=osd.3.0:277, at_version=63'79, trim_to=0'0, roll_forward_to=62'77))", 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.135308232, 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.70853546499999998, 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.706 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4099999999999999e-07 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511375+0000", 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7755000000000001e-05 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219030+0000", 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.70765444700000002 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219046+0000", 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6152999999999999e-05 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219182+0000", 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013656599999999999 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219190+0000", 2024-09-16T04:09:01.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1079999999999993e-06 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219848+0000", 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065702000000000002 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219882+0000", 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4875000000000001e-05 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:537 3.19s0 3:9fd10cc9:::smithi01639990-50:head [append 0~622592] snapc 13=[13,12,10] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:39.519316+0000", 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.127338870999999, 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016449781, 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 537 2024-09-16T04:09:01.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.519316+0000", 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.519316+0000", 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.519319+0000", 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2639999999999998e-06 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520728+0000", 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014093459999999999 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520743+0000", 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4647999999999999e-05 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520759+0000", 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6201999999999998e-05 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520815+0000", 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6076999999999999e-05 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520918+0000", 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000103041 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.522275+0000", 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013567500000000001 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.525027+0000", 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027523780000000002 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.535715+0000", 2024-09-16T04:09:01.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010687665000000001 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.535766+0000", 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1409999999999997e-05 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:538 3.19s0 3:9fd10cc9:::smithi01639990-50:head [append 0~442368] snapc 13=[13,12,10] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:39.522591+0000", 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.124063726999999, 2024-09-16T04:09:01.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013308851999999999, 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 538 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.522591+0000", 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.522591+0000", 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.522599+0000", 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.571e-06 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523302+0000", 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070285300000000003 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523316+0000", 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3647e-05 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523342+0000", 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6285e-05 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523403+0000", 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1705000000000006e-05 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523505+0000", 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010149099999999999 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.524458+0000", 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095253700000000005 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.526152+0000", 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001694152 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.535852+0000", 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097005860000000006 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.535900+0000", 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8025000000000002e-05 2024-09-16T04:09:01.721 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 67/56 ECSubWriteReply(tid=146, last_complete=67'24, committed=1, applied=1))", 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:42.548179+0000", 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.098475925999999, 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.18309741900000001, 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548179+0000", 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548179+0000", 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548180+0000", 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4199999999999995e-07 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548184+0000", 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4700000000000004e-06 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.730637+0000", 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.182453118 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.730647+0000", 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0650000000000005e-06 2024-09-16T04:09:01.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.730717+0000", 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0551999999999998e-05 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.731277+0000", 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055957200000000004 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds1 81/56 ECSubWrite(tid=221, reqid=osd.7.0:221, at_version=81'36, trim_to=0'0, roll_forward_to=80'34))", 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:56.485832+0000", 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.1608229359999997, 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67555770299999995, 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485832+0000", 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485832+0000", 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485833+0000", 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485854+0000", 2024-09-16T04:09:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1855999999999999e-05 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160608+0000", 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67475406599999999 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160614+0000", 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.789e-06 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160698+0000", 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3311000000000002e-05 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160705+0000", 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.161360+0000", 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065466099999999996 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.161390+0000", 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0108999999999999e-05 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:780 3.cs0 3:38c68ed3:::smithi01639990-42:head [copy-from ver 22] snapc 1d=[1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.488449+0000", 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.1582066209999997, 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.022970059000000001, 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.730 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 780 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488449+0000", 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488449+0000", 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488451+0000", 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7609999999999999e-06 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488464+0000", 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2547e-05 2024-09-16T04:09:01.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488485+0000", 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0789999999999999e-05 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488497+0000", 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919000000000001e-05 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488579+0000", 2024-09-16T04:09:01.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2398999999999995e-05 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488588+0000", 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7309999999999999e-06 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490560+0000", 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019719410000000001 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490628+0000", 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8275000000000006e-05 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.504399+0000", 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013771097 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506432+0000", 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020333460000000001 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.511365+0000", 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049331410000000003 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.511419+0000", 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3112000000000002e-05 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.736 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:781 3.cs0 3:38c68ed3:::smithi01639990-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.488524+0000", 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.1581310189999998, 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017905164000000001, 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 781 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488524+0000", 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488524+0000", 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488525+0000", 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6400000000000002e-07 2024-09-16T04:09:01.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488528+0000", 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2440000000000002e-06 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488537+0000", 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6209999999999993e-06 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488542+0000", 2024-09-16T04:09:01.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6890000000000001e-06 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488657+0000", 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011471799999999999 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488663+0000", 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1330000000000004e-06 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490738+0000", 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020745709999999999 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490765+0000", 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6897999999999999e-05 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506037+0000", 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015272819 2024-09-16T04:09:01.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506239+0000", 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00020141000000000001 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506429+0000", 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000190497 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:792 3.5s0 3:a7f99973:::smithi01639990-25:head [append 0~540672] snapc 1d=[1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.536981+0000", 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.1096739800000002, 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011312175000000001, 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 792 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.743 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.536981+0000", 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.536981+0000", 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.536983+0000", 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.328e-06 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.537799+0000", 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081688300000000002 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.537809+0000", 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2539999999999998e-06 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.537813+0000", 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5850000000000001e-06 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.537872+0000", 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8640000000000001e-05 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.537984+0000", 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112188 2024-09-16T04:09:01.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.538822+0000", 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083813799999999999 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.541701+0000", 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0028792969999999998 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.548241+0000", 2024-09-16T04:09:01.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0065395619999999996 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.548293+0000", 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2299999999999997e-05 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:817 3.cs0 3:338c6e39:::smithi01639990-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-16T04:09:01.748 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:58.504509+0000", 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.142145685, 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011997270000000001, 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 817 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504509+0000", 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504509+0000", 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504510+0000", 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8799999999999998e-07 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504511+0000", 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.361e-06 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504517+0000", 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1099999999999999e-06 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504521+0000", 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7799999999999998e-06 2024-09-16T04:09:01.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.507640+0000", 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00311883 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.507728+0000", 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7991999999999994e-05 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:01.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.515416+0000", 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0076882610000000001 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.515509+0000", 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3313000000000006e-05 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.516507+0000", 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00099733500000000002 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.753 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.754 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:01.754 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:01.754 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:01.755 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:09:01.755 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:09:01.756 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:09:01.756 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:09:01.757 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:09:01.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:09:01.753+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:09:01.856 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:09:01.855+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:09:01.900 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:01.900 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:01.901 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:09:01.901 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:01.909 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:09:01.957 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:09:01.956+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:09:02.053 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:02.054 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:02.054 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:02.054 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:02.054 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:02.058 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:02.057+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:02.062 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:09:02.159 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:02.157+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:02.166 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:02.166 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:02.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:02.166 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:02.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.166 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:09:02.166 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.713887176999997, 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091491550000000008, 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5800000000000001e-07 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450512+0000", 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057160000000000002 2024-09-16T04:09:02.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450541+0000", 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8835000000000001e-05 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450549+0000", 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3080000000000008e-06 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450607+0000", 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7763999999999998e-05 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450616+0000", 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459045+0000", 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084290709999999998 2024-09-16T04:09:02.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459089+0000", 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4233000000000002e-05 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:469 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~491520] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:02.171 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:09:02.172 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.611054213000003, 2024-09-16T04:09:02.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098323030000000006, 2024-09-16T04:09:02.172 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.172 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.172 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:02.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:02.172 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 469 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552774+0000", 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1999999999999999e-06 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553284+0000", 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00050997900000000003 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553297+0000", 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2723e-05 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553306+0000", 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5969999999999992e-06 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553363+0000", 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6922999999999997e-05 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553436+0000", 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3090000000000007e-05 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.554087+0000", 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065173499999999997 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.558220+0000", 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041326330000000001 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562517+0000", 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042965859999999998 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562605+0000", 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8837000000000003e-05 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:474 3.as0 3:5e52341f:::smithi01639990-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.603722251999997, 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097686969999999998, 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.178 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 474 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560106+0000", 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.63e-07 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560109+0000", 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.303e-06 2024-09-16T04:09:02.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560116+0000", 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3860000000000001e-06 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560122+0000", 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4450000000000004e-06 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563686+0000", 2024-09-16T04:09:02.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035639740000000001 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563776+0000", 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0278000000000005e-05 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568854+0000", 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0050781680000000001 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568970+0000", 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011579899999999999 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.569874+0000", 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00090388100000000004 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.726090556999999, 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015912599999999999, 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437757+0000", 2024-09-16T04:09:02.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9735e-05 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437829+0000", 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1605000000000003e-05 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437838+0000", 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442352+0000", 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045136799999999999 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442364+0000", 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2642e-05 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453618+0000", 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011254128 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453649+0000", 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1016000000000003e-05 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.725051948000001, 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014897297, 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438776+0000", 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3099999999999997e-07 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438797+0000", 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0596000000000001e-05 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438836+0000", 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9684000000000002e-05 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438845+0000", 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6479999999999992e-06 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442677+0000", 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038317239999999999 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442685+0000", 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453654+0000", 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010968637 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453673+0000", 2024-09-16T04:09:02.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8967999999999999e-05 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.723061998999999, 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012700487999999999, 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.193 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8900000000000001e-07 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440785+0000", 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9850000000000001e-05 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445485+0000", 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046992329999999997 2024-09-16T04:09:02.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445495+0000", 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0506000000000001e-05 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445668+0000", 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000172905 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445679+0000", 2024-09-16T04:09:02.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453438+0000", 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077588930000000002 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453466+0000", 2024-09-16T04:09:02.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8175e-05 2024-09-16T04:09:02.198 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.198 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.198 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.198 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=110, reqid=osd.5.0:109, at_version=0'0, trim_to=0'0, roll_forward_to=57'31))", 2024-09-16T04:09:02.198 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:02.199 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.699257546999998, 2024-09-16T04:09:02.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011518758, 2024-09-16T04:09:02.199 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.199 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.199 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.200 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:02.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:02.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8e-07 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464577+0000", 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9940000000000003e-06 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472531+0000", 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079544330000000003 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472538+0000", 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7789999999999998e-06 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472599+0000", 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0841999999999998e-05 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472610+0000", 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1185999999999999e-05 2024-09-16T04:09:02.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476060+0000", 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034496650000000002 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476089+0000", 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8679000000000001e-05 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.204 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.205 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=37, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:09:02.205 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:02.205 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.675754744999999, 2024-09-16T04:09:02.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.039596059000000003, 2024-09-16T04:09:02.205 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.205 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1e-07 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488075+0000", 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8789999999999999e-06 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526262+0000", 2024-09-16T04:09:02.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.038187556999999997 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526270+0000", 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3209999999999999e-06 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526424+0000", 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015469 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.527669+0000", 2024-09-16T04:09:02.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012444019999999999 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=38, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.633262468000002, 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028484058, 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.209 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530566+0000", 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.85e-07 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530570+0000", 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7319999999999998e-06 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558866+0000", 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028295515 2024-09-16T04:09:02.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558875+0000", 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7849999999999997e-06 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558966+0000", 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1611000000000001e-05 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.559049+0000", 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.263e-05 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 69/27 ECSubWrite(tid=153, reqid=osd.5.0:153, at_version=69'41, trim_to=0'0, roll_forward_to=68'39))", 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.62450905, 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0089824069999999995, 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.213 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:02.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539343+0000", 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4399000000000001e-05 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539420+0000", 2024-09-16T04:09:02.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7167000000000002e-05 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539428+0000", 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0679999999999997e-06 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539506+0000", 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7836000000000005e-05 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539518+0000", 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1627000000000001e-05 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548257+0000", 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087393379999999993 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548301+0000", 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3798999999999999e-05 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.217 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.14s5 69/48 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=69'49, trim_to=0'0, roll_forward_to=68'45))", 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.624464799999998, 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0093116210000000008, 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:02.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0999999999999998e-07 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539388+0000", 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4769999999999998e-05 2024-09-16T04:09:02.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539439+0000", 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1149000000000001e-05 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539450+0000", 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0557000000000001e-05 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539608+0000", 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000158679 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539620+0000", 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1260000000000001e-05 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548634+0000", 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009014256 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548674+0000", 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0240000000000001e-05 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.16s7 69/56 ECSubWrite(tid=152, reqid=osd.5.0:152, at_version=69'28, trim_to=0'0, roll_forward_to=68'26))", 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.539883+0000", 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.623944674000001, 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085560289999999997, 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539883+0000", 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539883+0000", 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539884+0000", 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189999999999999e-06 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539909+0000", 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5621000000000001e-05 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539956+0000", 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7209999999999997e-05 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539967+0000", 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0514e-05 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540036+0000", 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9221999999999996e-05 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540047+0000", 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0898e-05 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548370+0000", 2024-09-16T04:09:02.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0083228469999999995 2024-09-16T04:09:02.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548439+0000", 2024-09-16T04:09:02.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8697999999999999e-05 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds3 69/56 ECSubWrite(tid=201, reqid=client.4440.0:654, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.623262360999998, 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098306729999999998, 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6600000000000002e-07 2024-09-16T04:09:02.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540718+0000", 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015259 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540851+0000", 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013328999999999999 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540860+0000", 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7909999999999993e-06 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541117+0000", 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025658799999999998 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541127+0000", 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0834e-05 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.550371+0000", 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092436089999999999 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.550396+0000", 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4505e-05 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.231 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.12s6 69/56 ECSubWrite(tid=147, reqid=osd.0.0:147, at_version=69'84, trim_to=0'0, roll_forward_to=68'82))", 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.542608+0000", 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.621218889000001, 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021420679000000001, 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542608+0000", 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542608+0000", 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542609+0000", 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.15e-07 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542633+0000", 2024-09-16T04:09:02.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4542999999999999e-05 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542748+0000", 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000114911 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542756+0000", 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542862+0000", 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000105689 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542873+0000", 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0575e-05 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.563997+0000", 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021124446000000002 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564029+0000", 2024-09-16T04:09:02.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2101000000000003e-05 2024-09-16T04:09:02.236 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.236 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.236 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.236 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.12s6 69/56 ECSubWrite(tid=149, reqid=osd.0.0:149, at_version=69'86, trim_to=0'0, roll_forward_to=68'82))", 2024-09-16T04:09:02.236 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.544392+0000", 2024-09-16T04:09:02.236 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.619435295999999, 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019668749999999999, 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544392+0000", 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544392+0000", 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544393+0000", 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.195e-06 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544420+0000", 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6635000000000002e-05 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544463+0000", 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3479000000000002e-05 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544472+0000", 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8880000000000003e-06 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.551267+0000", 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00679526 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.551277+0000", 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2760000000000006e-06 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564037+0000", 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012760258 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564061+0000", 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3759000000000001e-05 2024-09-16T04:09:02.240 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds3 69/56 ECSubWrite(tid=203, reqid=osd.7.0:203, at_version=69'28, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.546802+0000", 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.617025194, 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088412939999999995, 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546802+0000", 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546802+0000", 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546802+0000", 2024-09-16T04:09:02.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.96e-07 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546839+0000", 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6677999999999997e-05 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546944+0000", 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010538400000000001 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546951+0000", 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3389999999999998e-06 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.547280+0000", 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00032936 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.547291+0000", 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1007e-05 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.555601+0000", 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0083102599999999999 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.555643+0000", 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.197e-05 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.246 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 69/56 ECSubWrite(tid=340, reqid=osd.3.0:340, at_version=69'98, trim_to=0'0, roll_forward_to=68'96))", 2024-09-16T04:09:02.246 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:02.246 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 17.609418568999999, 2024-09-16T04:09:02.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096797529999999993, 2024-09-16T04:09:02.246 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.246 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.247 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.247 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:02.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554437+0000", 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7382e-05 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554469+0000", 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2073000000000003e-05 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554480+0000", 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1358999999999999e-05 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554550+0000", 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9808999999999995e-05 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554564+0000", 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3634e-05 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564066+0000", 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095028419999999992 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564088+0000", 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1996000000000001e-05 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 74/56 ECSubWrite(tid=376, reqid=osd.3.0:376, at_version=74'104, trim_to=0'0, roll_forward_to=70'102))", 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:49.453728+0000", 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 12.710098816, 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092035429999999998, 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.251 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453728+0000", 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453728+0000", 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453729+0000", 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1e-07 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453787+0000", 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8541000000000001e-05 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453868+0000", 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1199000000000006e-05 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453875+0000", 2024-09-16T04:09:02.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8310000000000003e-06 2024-09-16T04:09:02.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453979+0000", 2024-09-16T04:09:02.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000103288 2024-09-16T04:09:02.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.255 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453988+0000", 2024-09-16T04:09:02.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.533e-06 2024-09-16T04:09:02.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.255 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.256 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.256 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.462890+0000", 2024-09-16T04:09:02.256 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0089015139999999993 2024-09-16T04:09:02.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.256 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.257 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.257 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.462932+0000", 2024-09-16T04:09:02.257 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2426999999999999e-05 2024-09-16T04:09:02.257 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.257 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.257 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.258 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.258 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:736 3.as0 3:5e52341f:::smithi01639990-38:head [copy-from ver 69] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:02.258 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439036+0000", 2024-09-16T04:09:02.258 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.7247913340000007, 2024-09-16T04:09:02.258 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.030166564999999999, 2024-09-16T04:09:02.259 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.259 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.259 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.259 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.259 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:02.259 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 736 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439036+0000", 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.260 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439036+0000", 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439038+0000", 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4339999999999999e-06 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439050+0000", 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1418e-05 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.261 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439070+0000", 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0251e-05 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439080+0000", 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4599999999999992e-06 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439154+0000", 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4820999999999994e-05 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.262 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439162+0000", 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6969999999999993e-06 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.443704+0000", 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045422580000000004 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.443816+0000", 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011184900000000001 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.263 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.461800+0000", 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017984236000000001 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.465505+0000", 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037041209999999999 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.264 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.469122+0000", 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036175270000000002 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.469203+0000", 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0493000000000003e-05 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.265 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:737 3.as0 3:5e52341f:::smithi01639990-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439252+0000", 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.7245751540000001, 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.024601304000000001, 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 737 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.266 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439252+0000", 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439252+0000", 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439253+0000", 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7899999999999998e-07 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.267 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439259+0000", 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3609999999999997e-06 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439269+0000", 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8430000000000005e-06 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439276+0000", 2024-09-16T04:09:02.268 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4649999999999997e-06 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439328+0000", 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1063999999999998e-05 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439334+0000", 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.269 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.443976+0000", 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046418049999999997 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.444023+0000", 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6872000000000001e-05 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.463639+0000", 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019616097999999998 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.270 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.463746+0000", 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010683600000000001 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.463853+0000", 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010751600000000001 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.271 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.272 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.272 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:02.273 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:09:02.274 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:02.271+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:02.374 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:02.373+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:02.417 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:02.417 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:09:02.417 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:02.426 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:09:02.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T04:09:02.475 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:02.474+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:02.514 INFO:tasks.rados.rados.0.smithi016.stdout:352: done (1 left) 2024-09-16T04:09:02.515 INFO:tasks.rados.rados.0.smithi016.stdout:353: read oid 23 snap -1 2024-09-16T04:09:02.515 INFO:tasks.rados.rados.0.smithi016.stdout:353: expect (ObjNum 137 snap 28 seq_num 137) 2024-09-16T04:09:02.515 INFO:tasks.rados.rados.0.smithi016.stdout:354: read oid 27 snap -1 2024-09-16T04:09:02.515 INFO:tasks.rados.rados.0.smithi016.stdout:354: expect (ObjNum 127 snap 23 seq_num 127) 2024-09-16T04:09:02.515 INFO:tasks.rados.rados.0.smithi016.stdout:355: read oid 47 snap -1 2024-09-16T04:09:02.515 INFO:tasks.rados.rados.0.smithi016.stdout:355: expect (ObjNum 128 snap 23 seq_num 128) 2024-09-16T04:09:02.515 INFO:tasks.rados.rados.0.smithi016.stdout:356: setattr oid 45 current snap is 31 2024-09-16T04:09:02.515 INFO:tasks.rados.rados.0.smithi016.stdout:351: expect (ObjNum 109 snap 18 seq_num 109) 2024-09-16T04:09:02.540 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:02.540 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:02.540 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:02.540 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:02.540 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:02.551 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:09:02.571 INFO:tasks.rados.rados.0.smithi016.stdout:354: expect (ObjNum 127 snap 23 seq_num 127) 2024-09-16T04:09:02.575 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:02.574+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:02.630 INFO:tasks.rados.rados.0.smithi016.stdout:355: expect (ObjNum 128 snap 23 seq_num 128) 2024-09-16T04:09:02.657 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:02.657 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:02.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:02.657 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 72.713995476999997, 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:09:02.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:09:02.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:09:02.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:09:02.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:02.664 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.731591234000007, 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:09:02.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:02.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:09:02.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:09:02.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:09:02.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:09:02.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 71.721228728, 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:09:02.672 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:09:02.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:09:02.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 70.224640676000007, 2024-09-16T04:09:02.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:09:02.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:09:02.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 69.770012121999997, 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:02.683 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:09:02.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:09:02.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:02.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.688 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 55.545743584, 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:02.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.692 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 54.967809623999997, 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:09:02.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.550996478999998, 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.697 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:02.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:09:02.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:09:02.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:09:02.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:09:02.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 53.549609064999999, 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.702 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.706 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.190290034999997, 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:09:02.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:09:02.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:09:02.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:09:02.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.217420085000001, 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019153804, 2024-09-16T04:09:02.712 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437299+0000", 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437320+0000", 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1121999999999999e-05 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437388+0000", 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7889e-05 2024-09-16T04:09:02.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437397+0000", 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4390000000000006e-06 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441658+0000", 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042607110000000004 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441667+0000", 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:09:02.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456416+0000", 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014748493 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456452+0000", 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6159999999999999e-05 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.214230374, 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017174697999999999, 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440489+0000", 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.015e-06 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440509+0000", 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0338000000000001e-05 2024-09-16T04:09:02.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440547+0000", 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440557+0000", 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7820000000000005e-06 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442068+0000", 2024-09-16T04:09:02.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015112179999999999 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442078+0000", 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4620000000000002e-06 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457639+0000", 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015560954 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457663+0000", 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4121000000000002e-05 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.as7 68/56 ECSubWrite(tid=57, reqid=osd.6.0:57, at_version=68'42, trim_to=0'0, roll_forward_to=64'40))", 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.120563122, 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017528866000000001, 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.721 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:02.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.49e-07 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534179+0000", 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3308000000000001e-05 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546651+0000", 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012471873 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546660+0000", 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2229999999999996e-06 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546730+0000", 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0123000000000006e-05 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546741+0000", 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0684e-05 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551646+0000", 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049059180000000004 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551684+0000", 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7588e-05 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 68/56 ECSubWriteReply(tid=200, last_complete=68'24, committed=1, applied=1))", 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:09:02.725 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 19.091551098, 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46574167999999999, 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:09:02.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563168+0000", 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4099999999999997e-07 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563174+0000", 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.327e-06 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028683+0000", 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46550948399999997 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028698+0000", 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4576000000000001e-05 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028858+0000", 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016062100000000001 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028909+0000", 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0730999999999999e-05 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:654 3.ds0 3:bc502819:::smithi01639990-5:head [create,setxattr XXWdXV_kiPaajZX\\PgmqNaZqSVmab^hrtRYeagZr (1894) in=40b,setxattr NdZr`e\\^kXiksh]\\XocenQrhYc]c`[proOfo`Pd^ (33) in=40b,setxattr _`NqSdldlQbUNl^ZNRba\\naZhhdraiOgpafrZb[o (103) in=40b,setxattr iadimrq_[jOPUblhnVqTYqQNedg (1647) in=27b,setxattr `e\\rtQi]nmoP\\cm]oRYN`[]^qVUTsU^_oXm]h\\ec (6452) in=40b,setxattr ppZXc_dYi_WWXcUdeqcpck_ceUVa^XftgnfgdP[q (3576) in=40b,setxattr hehZsP\\ajqhe^nfrZjQmsnfWoqq\\e\\NV^rOaZYtj (1787) in=40b,setxattr a]Smbg`^k`emWabQRYcpr_lQj`eh`[r\\riqa_sfd (1015) in=40b,setxattr eqsaZkZ[`t^R^grhYOVrVsspVQQp_Zo\\loUV[\\U[ (982) in=40b,setxattr Rn`k`cSYQj`XOZdn_mSPrqQtoUpWTk[^[t (1647) in=34b,setxattr ]TtVabQgUTabfmqae[UVe`cNsqqYfdh]\\sVoeXQr (383) in=40b,setxattr iRn][N_NY\\ZhmYdQ[[ptTP_NW^Z]_roZnOpcmR_Y (19016) in=40b,setxattr T_b_WhnZnkaaYkNmmOUnaZa`dhRhQSshYQVQnVql (13472) in=40b,setxattr \\hZXh`dT^W[c\\XodZZ_d\\QO\\\\[PVWthO_WVOPSQd (1681) in=40b,setxattr ctSOqNY_dUqTOjgXYjqgslgNRRPoT\\\\OXPPWrtt_ (522) in=40b,setxattr f\\hTOf^sbRaYeUntSV^WhUo]qV`bSNermptfXs]Q (1645) in=40b,setxattr ohNTdfibmYs`eUTiQa^bcYW\\aarhT]YQ_haagqpf (1119) in=40b,setxattr qqUtZZ]ataUj_[pnh\\jVtU_dRpi^g[VPNeS`Y\\hk (699) in=40b,setxattr qsgedRfqdW]]cYbWOUgh^ZleaSQiteejr^afXbUj (525) in=40b,setxattr [XmPqpoSg`fWjdlmQ_qabof]_WVcPaY[]k[ardgj (594) in=40b] snapc 15=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.120540063, 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019213674, 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.729 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 654 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534181+0000", 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5830000000000001e-06 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534398+0000", 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000216663 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534421+0000", 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3714e-05 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534434+0000", 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3185e-05 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534504+0000", 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9665000000000005e-05 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534510+0000", 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1709999999999999e-06 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.538902+0000", 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043914660000000001 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539044+0000", 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014215000000000001 2024-09-16T04:09:02.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540400+0000", 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013559189999999999 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541692+0000", 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001291964 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553277+0000", 2024-09-16T04:09:02.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011585432999999999 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553392+0000", 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011476099999999999 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:733 3.fs0 3:f7af954f:::smithi01639990-10:head [append 0~0] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.215908413999999, 2024-09-16T04:09:02.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016112728999999999, 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 733 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438813+0000", 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6089999999999999e-06 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438822+0000", 2024-09-16T04:09:02.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6199999999999994e-06 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438844+0000", 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1933000000000002e-05 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438856+0000", 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1588e-05 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438966+0000", 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011003899999999999 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438974+0000", 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5019999999999992e-06 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.446497+0000", 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075231389999999999 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.446592+0000", 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4358999999999996e-05 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.447539+0000", 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00094693900000000005 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454766+0000", 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0072270520000000003 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454838+0000", 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1970000000000004e-05 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454923+0000", 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4979000000000001e-05 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:734 3.fs0 3:f7af954f:::smithi01639990-10:head [rollback 1a] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:02.742 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.215834361000001, 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019165187, 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 734 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:02.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7399999999999998e-07 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438890+0000", 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8889999999999999e-06 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438903+0000", 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2466000000000001e-05 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438909+0000", 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3640000000000004e-06 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439564+0000", 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000655084 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439571+0000", 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.472e-06 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448110+0000", 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085393550000000002 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448181+0000", 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1259000000000003e-05 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448725+0000", 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00054394000000000003 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.457803+0000", 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0090773959999999997 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.457955+0000", 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015255800000000001 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458049+0000", 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3930000000000004e-05 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.748 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:735 3.fs0 3:f7af954f:::smithi01639990-10:head [append 0~0] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.215132273, 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019107652999999999, 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 735 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439587+0000", 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439591+0000", 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2570000000000001e-06 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439607+0000", 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5846e-05 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439614+0000", 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6579999999999996e-06 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439657+0000", 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3402999999999998e-05 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439663+0000", 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0959999999999997e-06 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449536+0000", 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098724250000000006 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449600+0000", 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4857999999999995e-05 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449969+0000", 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036863400000000001 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458052+0000", 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0080833569999999993 2024-09-16T04:09:02.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458113+0000", 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0764999999999998e-05 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458694+0000", 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00058057699999999998 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 81/27 ECSubWriteReply(tid=222, last_complete=81'16, committed=1, applied=1))", 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 6.1562247379999997, 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.59824786600000002, 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9100000000000003e-07 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498500+0000", 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3290000000000001e-06 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096610+0000", 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.59810986399999999 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096617+0000", 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9609999999999999e-06 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096710+0000", 2024-09-16T04:09:02.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3670000000000003e-05 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096741+0000", 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1251000000000003e-05 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 81/56 ECSubWriteReply(tid=223, last_complete=81'36, committed=1, applied=1))", 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.162644+0000", 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.4920740889999999, 2024-09-16T04:09:02.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.32306014599999999, 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162644+0000", 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162644+0000", 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:02.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162645+0000", 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3699999999999999e-07 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162658+0000", 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3112000000000001e-05 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485612+0000", 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.32295339699999998 2024-09-16T04:09:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485618+0000", 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4489999999999996e-06 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485676+0000", 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7414999999999998e-05 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485704+0000", 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8836e-05 2024-09-16T04:09:02.761 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.762 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.762 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.762 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:02.762 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:02.762 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:02.763 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:09:02.764 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:09:02.761+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:09:02.863 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:02.862+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:02.869 INFO:tasks.rados.rados.0.smithi016.stdout:353: expect (ObjNum 137 snap 28 seq_num 137) 2024-09-16T04:09:02.870 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:02.871 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:02.871 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:09:02.871 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:02.880 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:09:02.965 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:09:02.963+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:09:02.982 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:02.982 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:02.983 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:09:02.983 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:09:02.983 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:02.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:09:03.001 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 45 v 51 (ObjNum 86 snap 11 seq_num 86) dirty exists 2024-09-16T04:09:03.002 INFO:tasks.rados.rados.0.smithi016.stdout:351: done (4 left) 2024-09-16T04:09:03.002 INFO:tasks.rados.rados.0.smithi016.stdout:353: done (3 left) 2024-09-16T04:09:03.002 INFO:tasks.rados.rados.0.smithi016.stdout:354: done (2 left) 2024-09-16T04:09:03.002 INFO:tasks.rados.rados.0.smithi016.stdout:355: done (1 left) 2024-09-16T04:09:03.002 INFO:tasks.rados.rados.0.smithi016.stdout:356: done (0 left) 2024-09-16T04:09:03.002 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 41 current snap is 31 2024-09-16T04:09:03.002 INFO:tasks.rados.rados.0.smithi016.stdout:357: seq_num 146 ranges {1720320=49152} 2024-09-16T04:09:03.011 INFO:tasks.rados.rados.0.smithi016.stdout:357: writing smithi01639990-41 from 1720320 to 1769472 tid 1 2024-09-16T04:09:03.011 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 50 current snap is 31 2024-09-16T04:09:03.011 INFO:tasks.rados.rados.0.smithi016.stdout:358: seq_num 147 ranges {1523712=65536} 2024-09-16T04:09:03.020 INFO:tasks.rados.rados.0.smithi016.stdout:358: writing smithi01639990-50 from 1523712 to 1589248 tid 1 2024-09-16T04:09:03.020 INFO:tasks.rados.rados.0.smithi016.stdout:359: snap_create 2024-09-16T04:09:03.020 INFO:tasks.rados.rados.0.smithi016.stdout:357: finishing write tid 1 to smithi01639990-41 2024-09-16T04:09:03.020 INFO:tasks.rados.rados.0.smithi016.stdout:357: finishing write tid 2 to smithi01639990-41 2024-09-16T04:09:03.020 INFO:tasks.rados.rados.0.smithi016.stdout:357: finishing write tid 3 to smithi01639990-41 2024-09-16T04:09:03.021 INFO:tasks.rados.rados.0.smithi016.stdout:357: oid 41 updating version 0 to 113 2024-09-16T04:09:03.021 INFO:tasks.rados.rados.0.smithi016.stdout:357: oid 41 version 113 is already newer than 112 2024-09-16T04:09:03.021 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 41 v 113 (ObjNum 146 snap 31 seq_num 146) dirty exists 2024-09-16T04:09:03.021 INFO:tasks.rados.rados.0.smithi016.stdout:357: left oid 41 (ObjNum 146 snap 31 seq_num 146) 2024-09-16T04:09:03.024 INFO:tasks.rados.rados.0.smithi016.stdout:358: finishing write tid 1 to smithi01639990-50 2024-09-16T04:09:03.065 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:09:03.064+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:09:03.097 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 53.953359996000003, 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.099 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:09:03.100 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.101 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:09:03.102 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.635304388000002, 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.104 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.105 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:09:03.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:09:03.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.106 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.106 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.106 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:09:03.106 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:09:03.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.107 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.107 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.504762229000001, 2024-09-16T04:09:03.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:09:03.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:09:03.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 40.423839676, 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.113 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:03.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:03.117 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 40.263435618000003, 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:09:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.121 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.587330780999999, 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:09:03.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.565617756000002, 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:09:03.127 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:09:03.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:09:03.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.565565528999997, 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:03.133 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:09:03.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:09:03.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.11s6 68/56 ECSubWrite(tid=17, reqid=osd.10.0:17, at_version=68'19, trim_to=0'0, roll_forward_to=64'17))", 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.555879607000001, 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014242092, 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:03.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8500000000000001e-07 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539199+0000", 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4397999999999999e-05 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539248+0000", 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9033000000000003e-05 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539258+0000", 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8800000000000003e-06 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539858+0000", 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060046199999999996 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539867+0000", 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.363e-06 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553388+0000", 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013520249 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553416+0000", 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8121999999999999e-05 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.142 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs3 68/56 ECSubWrite(tid=327, reqid=osd.3.0:327, at_version=68'66, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.555588148999998, 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012797307000000001, 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539465+0000", 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539466+0000", 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4000000000000001e-07 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539484+0000", 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8391999999999999e-05 2024-09-16T04:09:03.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539516+0000", 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1439999999999997e-05 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539524+0000", 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4570000000000005e-06 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542244+0000", 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027199070000000001 2024-09-16T04:09:03.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542254+0000", 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8500000000000006e-06 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552231+0000", 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0099774170000000006 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552262+0000", 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1204000000000001e-05 2024-09-16T04:09:03.146 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=64, reqid=osd.4.0:64, at_version=68'43, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.554765603, 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014089379000000001, 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1500000000000001e-07 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540308+0000", 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0585999999999999e-05 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540337+0000", 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8560000000000001e-05 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540345+0000", 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6299999999999998e-06 2024-09-16T04:09:03.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540400+0000", 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4818000000000003e-05 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540411+0000", 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1009000000000001e-05 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554349+0000", 2024-09-16T04:09:03.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013938394 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554377+0000", 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8167e-05 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=66, reqid=osd.4.0:66, at_version=68'45, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:09:03.151 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.552924312000002, 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01391278, 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542130+0000", 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6600000000000005e-07 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542154+0000", 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3862999999999998e-05 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542183+0000", 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9108999999999999e-05 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542192+0000", 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549962+0000", 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007769977 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549972+0000", 2024-09-16T04:09:03.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0074e-05 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556017+0000", 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060444319999999998 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556042+0000", 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5012999999999999e-05 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s7 68/56 ECSubWrite(tid=197, reqid=osd.7.0:197, at_version=68'24, trim_to=0'0, roll_forward_to=64'22))", 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.552580877, 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013604978, 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542472+0000", 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542473+0000", 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5099999999999999e-07 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542491+0000", 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8417e-05 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542561+0000", 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9744000000000002e-05 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542568+0000", 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.289e-06 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546547+0000", 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0039792350000000002 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.546557+0000", 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8870000000000004e-06 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556052+0000", 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0094947169999999997 2024-09-16T04:09:03.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556077+0000", 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5537999999999998e-05 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=119, reqid=client.4440.0:705, at_version=73'60, trim_to=0'0, roll_forward_to=73'57))", 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.600368237, 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035584957, 2024-09-16T04:09:03.160 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494686+0000", 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9999999999999996e-07 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494705+0000", 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9208999999999999e-05 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529770+0000", 2024-09-16T04:09:03.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035064924999999997 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529777+0000", 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5069999999999997e-06 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529831+0000", 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4122000000000001e-05 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529845+0000", 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4571e-05 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530243+0000", 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039740100000000003 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530270+0000", 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7421999999999999e-05 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.164 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:708 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 69] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.583298697, 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49730772000000001, 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:03.165 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 708 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511756+0000", 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6029999999999999e-06 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511762+0000", 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2500000000000003e-06 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511774+0000", 2024-09-16T04:09:03.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1761e-05 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511797+0000", 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2416999999999999e-05 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511872+0000", 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5437000000000003e-05 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:03.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511975+0000", 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010341899999999999 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.526578+0000", 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014602084 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529422+0000", 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028444880000000001 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008984+0000", 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47956190999999998 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009062+0000", 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8350999999999996e-05 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:709 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.583243195, 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017140263999999999, 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 709 2024-09-16T04:09:03.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0800000000000001e-07 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511812+0000", 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7799999999999999e-06 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511816+0000", 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.172e-06 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511819+0000", 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9739999999999998e-06 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512187+0000", 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036796799999999998 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512240+0000", 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3013999999999998e-05 2024-09-16T04:09:03.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528792+0000", 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016551461999999999 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528863+0000", 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0797000000000007e-05 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528950+0000", 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7788999999999996e-05 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.564426679, 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.36295003300000001, 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.176 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7899999999999998e-07 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530637+0000", 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892881+0000", 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.362243749 2024-09-16T04:09:03.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892890+0000", 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5620000000000004e-06 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892950+0000", 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0270999999999999e-05 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892959+0000", 2024-09-16T04:09:03.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1689999999999998e-06 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893539+0000", 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000579628 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893577+0000", 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8056999999999998e-05 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.180 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=131, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.564142142, 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47861527599999998, 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:09:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530912+0000", 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4400000000000001e-07 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530915+0000", 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5530000000000001e-06 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008849+0000", 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47793379000000002 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008854+0000", 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0340000000000004e-06 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008927+0000", 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3088000000000002e-05 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009526+0000", 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059936699999999996 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=132, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:09:03.184 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 14.084722872, 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44531548500000001, 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010331+0000", 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9600000000000001e-07 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010332+0000", 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.454e-06 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455554+0000", 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44522236199999998 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455561+0000", 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4189999999999999e-06 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455608+0000", 2024-09-16T04:09:03.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6928000000000002e-05 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455646+0000", 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8126000000000001e-05 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.188 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:819 3.9s0 3:94de6fe1:::smithi01639990-3:head [stat,delete] snapc 1e=[1e,1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:58.510155+0000", 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 4.5848982019999998, 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016397412, 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 819 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510155+0000", 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510155+0000", 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510157+0000", 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.823e-06 2024-09-16T04:09:03.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510167+0000", 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0237e-05 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510184+0000", 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7007000000000002e-05 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510193+0000", 2024-09-16T04:09:03.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5050000000000007e-06 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510261+0000", 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8507e-05 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510270+0000", 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8890000000000008e-06 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.516074+0000", 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058035379999999996 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.516179+0000", 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010501 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.518371+0000", 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021920770000000002 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.524696+0000", 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063251929999999998 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.526496+0000", 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001800486 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.526553+0000", 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6140000000000001e-05 2024-09-16T04:09:03.194 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.195 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.195 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.195 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.195 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.195 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:03.196 INFO:tasks.rados.rados.0.smithi016.stdout:358: finishing write tid 2 to smithi01639990-50 2024-09-16T04:09:03.196 INFO:tasks.rados.rados.0.smithi016.stdout:358: finishing write tid 3 to smithi01639990-50 2024-09-16T04:09:03.197 INFO:tasks.rados.rados.0.smithi016.stdout:358: oid 50 updating version 0 to 50 2024-09-16T04:09:03.197 INFO:tasks.rados.rados.0.smithi016.stdout:358: oid 50 updating version 50 to 51 2024-09-16T04:09:03.197 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 50 v 51 (ObjNum 147 snap 31 seq_num 147) dirty exists 2024-09-16T04:09:03.197 INFO:tasks.rados.rados.0.smithi016.stdout:358: left oid 50 (ObjNum 147 snap 31 seq_num 147) 2024-09-16T04:09:03.197 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:09:03.198 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:09:03.195+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:09:03.296 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:03.295+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:03.343 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:03.343 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:03.343 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:09:03.343 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:03.351 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:09:03.398 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:03.396+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:03.493 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:03.493 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:03.493 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:09:03.493 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:09:03.493 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:03.499 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:03.497+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:03.501 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:09:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2024-09-16T04:09:03.520 INFO:tasks.rados.rados.0.smithi016.stdout:357: done (2 left) 2024-09-16T04:09:03.520 INFO:tasks.rados.rados.0.smithi016.stdout:358: done (1 left) 2024-09-16T04:09:03.520 INFO:tasks.rados.rados.0.smithi016.stdout:359: done (0 left) 2024-09-16T04:09:03.520 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 33 current snap is 32 2024-09-16T04:09:03.520 INFO:tasks.rados.rados.0.smithi016.stdout:360: seq_num 148 ranges {2326528=81920} 2024-09-16T04:09:03.532 INFO:tasks.rados.rados.0.smithi016.stdout:360: writing smithi01639990-33 from 2326528 to 2408448 tid 1 2024-09-16T04:09:03.532 INFO:tasks.rados.rados.0.smithi016.stdout:361: snap_remove snap 25 2024-09-16T04:09:03.600 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:03.598+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 56.549131854000002, 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.647 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:09:03.648 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.649 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.650 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.213614884000002, 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.651 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.652 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.653 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.654 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 44.753664657000002, 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.655 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:09:03.656 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.657 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.658 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 40.430152051999997, 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.659 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.660 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.661 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:09:03.662 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 48/46 ECSubWrite(tid=194, reqid=client.4440.0:381, at_version=48'54, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 40.172368194000001, 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078488229999999996, 2024-09-16T04:09:03.663 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471071+0000", 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.664 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471072+0000", 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0279999999999999e-06 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471418+0000", 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00034541800000000001 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471457+0000", 2024-09-16T04:09:03.665 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471492+0000", 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5018000000000003e-05 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471798+0000", 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00030615599999999997 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.471812+0000", 2024-09-16T04:09:03.666 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.448e-05 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478881+0000", 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070685549999999998 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.478920+0000", 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9072000000000001e-05 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.667 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.cs4 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 31.203253726, 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0084259260000000002, 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.668 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440186+0000", 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.91e-07 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.669 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440211+0000", 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4978999999999999e-05 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440257+0000", 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6569000000000001e-05 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440268+0000", 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1056000000000001e-05 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.670 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440348+0000", 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9795000000000004e-05 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.440364+0000", 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5996999999999999e-05 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448570+0000", 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082059590000000005 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.671 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.448611+0000", 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.138e-05 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.bs7 57/56 [PushOp(3:d683cb60:::smithi01639990-49:head, version: 53'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d683cb60:::smithi01639990-49:head@53'38, size: 2310144, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 31.082671555000001, 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.043702234999999999, 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.672 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560769+0000", 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0049999999999999e-06 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.561301+0000", 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053276500000000004 2024-09-16T04:09:03.674 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603955+0000", 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042653605999999997 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603967+0000", 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2065000000000001e-05 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604063+0000", 2024-09-16T04:09:03.675 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5599000000000004e-05 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604470+0000", 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040719500000000001 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=252, reqid=osd.3.0:252, at_version=57'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:03.676 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.733945415000001, 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27497091299999998, 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:03.677 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909517+0000", 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2886000000000001e-05 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.678 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183847+0000", 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27432981400000001 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183853+0000", 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2929999999999997e-06 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183902+0000", 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9008999999999998e-05 2024-09-16T04:09:03.679 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183911+0000", 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184435+0000", 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052493699999999995 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184465+0000", 2024-09-16T04:09:03.680 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9292e-05 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=253, reqid=osd.3.0:253, at_version=57'67, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.733317383999999, 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50851210899999999, 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.681 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910123+0000", 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:09:03.682 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910136+0000", 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.331e-05 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416910+0000", 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50677369000000005 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.683 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416919+0000", 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8290000000000003e-06 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416975+0000", 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5983999999999998e-05 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416985+0000", 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418599+0000", 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001614515 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418634+0000", 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4588999999999997e-05 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.685 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 64/56 ECSubWrite(tid=285, reqid=client.4440.0:535, at_version=0'0, trim_to=0'0, roll_forward_to=64'72))", 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.101870314999999, 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012095003, 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.686 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541570+0000", 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.173e-06 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541582+0000", 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2106999999999999e-05 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552877+0000", 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011294692 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552889+0000", 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1875000000000001e-05 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552958+0000", 2024-09-16T04:09:03.688 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9510000000000004e-05 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552967+0000", 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6240000000000008e-06 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553631+0000", 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066445299999999996 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.689 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553664+0000", 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2568999999999999e-05 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 68/46 ECSubWrite(tid=65, reqid=osd.4.0:65, at_version=68'22, trim_to=0'0, roll_forward_to=67'20))", 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.102144057, 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010787494999999999, 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.690 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.691 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541313+0000", 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7431e-05 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541369+0000", 2024-09-16T04:09:03.692 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6311999999999999e-05 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541376+0000", 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5729999999999996e-06 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544486+0000", 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031104150000000001 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.693 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544495+0000", 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0960000000000007e-06 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552010+0000", 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075148200000000002 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552083+0000", 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2661000000000001e-05 2024-09-16T04:09:03.694 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 68/56 ECSubWrite(tid=329, reqid=osd.3.0:329, at_version=68'68, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.100690337, 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093665009999999993, 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.695 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542749+0000", 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542750+0000", 2024-09-16T04:09:03.696 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.23e-07 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542770+0000", 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0679999999999999e-05 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542842+0000", 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1391999999999997e-05 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.697 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542850+0000", 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2320000000000001e-06 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545061+0000", 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022114220000000002 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.545070+0000", 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6780000000000006e-06 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.698 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552088+0000", 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070176939999999997 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552115+0000", 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7679999999999999e-05 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds2 69/56 ECSubWrite(tid=201, reqid=client.4440.0:654, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:03.699 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.102852472999999, 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0090194439999999997, 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.700 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540587+0000", 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1400000000000001e-07 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540777+0000", 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000189886 2024-09-16T04:09:03.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540803+0000", 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6466e-05 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540811+0000", 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0730000000000005e-06 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540879+0000", 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8003999999999995e-05 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.540889+0000", 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8460000000000003e-06 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.549571+0000", 2024-09-16T04:09:03.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0086822059999999996 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.549606+0000", 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4749000000000002e-05 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=336, reqid=osd.3.0:336, at_version=69'70, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:09:03.704 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.099746292999999, 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0077691619999999996, 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:09:03.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543693+0000", 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.05e-07 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543708+0000", 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4431e-05 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543897+0000", 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018918 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543906+0000", 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3349999999999996e-06 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543954+0000", 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7793999999999997e-05 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.543963+0000", 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4259999999999992e-06 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551434+0000", 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0074706800000000004 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551462+0000", 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8110999999999999e-05 2024-09-16T04:09:03.708 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=338, reqid=osd.3.0:338, at_version=69'72, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.099052108999999, 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010064081000000001, 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544388+0000", 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9800000000000003e-07 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544409+0000", 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0993e-05 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544442+0000", 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2937e-05 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544449+0000", 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.588e-06 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547347+0000", 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028976599999999998 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547357+0000", 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6479999999999998e-06 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554419+0000", 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070620819999999999 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554451+0000", 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2475000000000003e-05 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s6 69/56 ECSubWrite(tid=202, reqid=osd.7.0:202, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.097715313999998, 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010752746000000001, 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.713 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545725+0000", 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4199999999999995e-07 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545750+0000", 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4915000000000001e-05 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546880+0000", 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011307870000000001 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546891+0000", 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0825000000000001e-05 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551495+0000", 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004604131 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551504+0000", 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6230000000000003e-06 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556450+0000", 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049460210000000001 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556477+0000", 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6801999999999999e-05 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.717 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.ds2 69/56 ECSubWrite(tid=203, reqid=osd.7.0:203, at_version=69'28, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.097686278000001, 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078627160000000005, 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.718 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545753+0000", 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.91e-07 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545769+0000", 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5562000000000001e-05 2024-09-16T04:09:03.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546923+0000", 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001154457 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546936+0000", 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3183000000000001e-05 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547734+0000", 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00079775500000000004 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547744+0000", 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5489999999999995e-06 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.553577+0000", 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058334809999999997 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.553616+0000", 2024-09-16T04:09:03.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8438e-05 2024-09-16T04:09:03.722 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.722 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.722 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.723 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 82/56 ECSubWrite(tid=197, reqid=client.4440.0:785, at_version=82'40, trim_to=0'0, roll_forward_to=74'38))", 2024-09-16T04:09:03.723 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.1506726770000002, 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0095796900000000001, 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6400000000000001e-07 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492863+0000", 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6371000000000003e-05 2024-09-16T04:09:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492883+0000", 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9507000000000001e-05 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492894+0000", 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0998999999999999e-05 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492958+0000", 2024-09-16T04:09:03.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3898000000000004e-05 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492968+0000", 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0012e-05 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.502307+0000", 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093395509999999998 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.502346+0000", 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9187999999999998e-05 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s5 82/56 ECSubWrite(tid=203, reqid=client.4440.0:808, at_version=82'31, trim_to=0'0, roll_forward_to=80'29))", 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.576184+0000", 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.067255007, 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0087388190000000001, 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.576184+0000", 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.576184+0000", 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.576185+0000", 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.06e-07 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.579980+0000", 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.003795245 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.580013+0000", 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2743000000000001e-05 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.580021+0000", 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8399999999999995e-06 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.580088+0000", 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7337999999999998e-05 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.580100+0000", 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2335e-05 2024-09-16T04:09:03.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.584876+0000", 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0047755929999999999 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.584923+0000", 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7018999999999998e-05 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.17s2 83/56 ECSubWrite(tid=206, reqid=client.4440.0:818, at_version=0'0, trim_to=0'0, roll_forward_to=83'73))", 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:58.510113+0000", 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.1333262240000002, 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.57355782399999999, 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510113+0000", 2024-09-16T04:09:03.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510113+0000", 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510114+0000", 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.68e-07 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510124+0000", 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0117e-05 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082885+0000", 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.57276119999999997 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082895+0000", 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7580000000000004e-06 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082959+0000", 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4545e-05 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082968+0000", 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9250000000000001e-06 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.083638+0000", 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00067024900000000004 2024-09-16T04:09:03.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.083671+0000", 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2462000000000002e-05 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:03.737 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:03.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:09:03.739 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:03.737+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:03.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:09:03.840 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:09:03.838+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:09:03.883 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:03.883 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:09:03.883 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:03.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:09:03.904 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:09:03.904 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:09:03.926 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:09:03.926 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:09:03.940 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:09:03.938+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:09:03.946 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:09:03.947 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:09:03.968 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:09:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:09:03.990 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:09:03.990 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:09:03.994 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:09:03.994 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:09:03.994 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:04.003 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:09:04.012 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:09:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:09:04.033 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:09:04.034 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:09:04.041 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:04.040+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:04.055 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:09:04.055 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:09:04.076 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:09:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:09:04.097 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:09:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:09:04.108 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:04.108 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:09:04.108 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:09:04.108 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:09:04.108 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.108 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 52/46 ECSubWrite(tid=210, reqid=client.4440.0:434, at_version=52'63, trim_to=0'0, roll_forward_to=50'60))", 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 36.588055734999998, 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83621405000000004, 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.109 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8200000000000001e-07 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517790+0000", 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2540000000000001e-05 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.110 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352789+0000", 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83499984500000002 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352804+0000", 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4569999999999999e-05 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352928+0000", 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012396799999999999 2024-09-16T04:09:04.111 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352943+0000", 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353943+0000", 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099977500000000006 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353981+0000", 2024-09-16T04:09:04.112 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7700999999999997e-05 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=220, reqid=client.4440.0:452, at_version=53'61, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 35.586052182000003, 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51827822899999998, 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.113 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519771+0000", 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.61e-07 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.114 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519788+0000", 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6699999999999999e-05 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036882+0000", 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51709414300000001 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036895+0000", 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3363999999999999e-05 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.115 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036964+0000", 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8734999999999998e-05 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036973+0000", 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0289999999999994e-06 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038014+0000", 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010410529999999999 2024-09-16T04:09:04.116 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038049+0000", 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4443999999999998e-05 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:526 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~622592] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.475255793999999, 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01502261, 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.117 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 526 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.118 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630568+0000", 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5680000000000001e-06 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631353+0000", 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078490700000000005 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.119 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631367+0000", 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3441000000000001e-05 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631375+0000", 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8809999999999996e-06 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631465+0000", 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0673000000000004e-05 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.120 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631565+0000", 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9753000000000005e-05 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.632641+0000", 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001076097 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634418+0000", 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017769680000000001 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.121 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645537+0000", 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011119393999999999 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645589+0000", 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1928000000000002e-05 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.122 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:527 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.474418720999999, 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015053211, 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 527 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.123 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631405+0000", 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.08e-06 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.124 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631411+0000", 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9009999999999999e-06 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631419+0000", 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631426+0000", 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4999999999999996e-06 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.125 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633558+0000", 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002131796 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633663+0000", 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105833 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634106+0000", 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00044296899999999998 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.126 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.636380+0000", 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022740849999999999 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646383+0000", 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01000238 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646457+0000", 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4258000000000006e-05 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:528 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.474344139999999, 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65958352099999995, 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.128 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 528 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.129 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631479+0000", 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9599999999999998e-07 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631483+0000", 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8160000000000004e-06 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631494+0000", 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0407999999999999e-05 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.130 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631501+0000", 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2269999999999996e-06 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634655+0000", 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031544339999999998 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634732+0000", 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6495999999999994e-05 2024-09-16T04:09:04.131 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646864+0000", 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012131807999999999 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646965+0000", 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010173 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.291062+0000", 2024-09-16T04:09:04.132 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64409670600000002 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:695 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~589824] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.656796148, 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014512838, 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.133 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 695 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.134 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449028+0000", 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.917e-06 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450009+0000", 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000980367 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450023+0000", 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.416e-05 2024-09-16T04:09:04.135 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450032+0000", 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5820000000000008e-06 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450094+0000", 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1698000000000005e-05 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450210+0000", 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116104 2024-09-16T04:09:04.136 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451973+0000", 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001762958 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463392+0000", 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011419024999999999 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463450+0000", 2024-09-16T04:09:04.137 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8251999999999997e-05 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463539+0000", 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8775000000000003e-05 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:696 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~753664] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.653663714, 2024-09-16T04:09:04.138 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013764281, 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 696 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.139 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452161+0000", 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2579999999999998e-06 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453157+0000", 2024-09-16T04:09:04.140 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099543299999999999 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453172+0000", 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5917e-05 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453182+0000", 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3959999999999995e-06 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455292+0000", 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021103379999999998 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455406+0000", 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000114203 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456783+0000", 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001376377 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.142 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463544+0000", 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067608640000000001 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465825+0000", 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022816350000000002 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465923+0000", 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7860000000000002e-05 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.143 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:700 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.634513205999999, 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014504263, 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.144 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 700 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.145 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471310+0000", 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.08e-07 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471312+0000", 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0540000000000002e-06 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471318+0000", 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5080000000000002e-06 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471322+0000", 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3629999999999998e-06 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472045+0000", 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00072327100000000005 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472133+0000", 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8528000000000003e-05 2024-09-16T04:09:04.147 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484130+0000", 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011996163000000001 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484213+0000", 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3800000000000004e-05 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.485814+0000", 2024-09-16T04:09:04.148 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016001679999999999 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.575188311, 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095992542, 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:04.149 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530636+0000", 2024-09-16T04:09:04.150 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.522e-06 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530651+0000", 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4980999999999999e-05 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625860+0000", 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095209590999999996 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625873+0000", 2024-09-16T04:09:04.151 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2413e-05 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625978+0000", 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010492099999999999 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625994+0000", 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6058000000000001e-05 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:04.152 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626583+0000", 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058931700000000001 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626627+0000", 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3739000000000003e-05 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:803 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=64b] snapc 1d=[1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:09:04.153 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.5436501319999998, 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.77068146599999998, 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 803 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.154 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562173+0000", 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3200000000000005e-07 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562176+0000", 2024-09-16T04:09:04.155 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9900000000000002e-06 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562183+0000", 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9480000000000002e-06 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562188+0000", 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1479999999999997e-06 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.156 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.565211+0000", 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030228519999999999 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.565509+0000", 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000297578 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.566353+0000", 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00084484499999999999 2024-09-16T04:09:04.157 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.568092+0000", 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017385720000000001 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.332782+0000", 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.76468974099999998 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.332854+0000", 2024-09-16T04:09:04.158 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.216e-05 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:804 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.5436265809999998, 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4794380249999999, 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.159 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 804 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.160 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2e-07 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562199+0000", 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1119999999999999e-06 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562206+0000", 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9369999999999998e-06 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.161 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562211+0000", 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2569999999999998e-06 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.567444+0000", 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005233265 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.567504+0000", 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9769999999999999e-05 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.162 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.333251+0000", 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.76574722100000003 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.333355+0000", 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010408600000000001 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041634+0000", 2024-09-16T04:09:04.163 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.70827885700000004 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 82/56 ECSubWriteReply(tid=41, last_complete=82'37, committed=1, applied=1))", 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.572277+0000", 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.533545674, 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.754736132, 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572277+0000", 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572277+0000", 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572278+0000", 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5199999999999996e-07 2024-09-16T04:09:04.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572281+0000", 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7290000000000002e-06 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.326881+0000", 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.75459961200000003 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.326897+0000", 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6180000000000001e-05 2024-09-16T04:09:04.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.326988+0000", 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0477999999999997e-05 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.327013+0000", 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5480999999999999e-05 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpReadReply(3.4s0 82/56 ECSubReadReply(tid=43, attrs_read=0))", 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:58.334354+0000", 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.7714688000000001, 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.70728895800000002, 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.334354+0000", 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.334354+0000", 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.334354+0000", 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.06e-07 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.334369+0000", 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5041e-05 2024-09-16T04:09:04.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041321+0000", 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.70695176299999996 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041348+0000", 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6659999999999999e-05 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041459+0000", 2024-09-16T04:09:04.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011134200000000001 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041643+0000", 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018354600000000001 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:848 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~4063232,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.513588+0000", 2024-09-16T04:09:04.171 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.592234221, 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.029163938, 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 848 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513588+0000", 2024-09-16T04:09:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513588+0000", 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513590+0000", 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8470000000000001e-06 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513600+0000", 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3619999999999995e-06 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513616+0000", 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6758000000000001e-05 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513633+0000", 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7139999999999999e-05 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513708+0000", 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4762e-05 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513717+0000", 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4870000000000002e-06 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520258+0000", 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065412760000000004 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520352+0000", 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3752000000000006e-05 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.542752+0000", 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022400554 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:849 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~4063232,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:04.176 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.513662+0000", 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.5921607610000001, 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.032938179999999997, 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 849 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513662+0000", 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513662+0000", 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513663+0000", 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513668+0000", 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.835e-06 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513677+0000", 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6810000000000004e-06 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513685+0000", 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3280000000000006e-06 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514004+0000", 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000318146 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514011+0000", 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3599999999999998e-06 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520894+0000", 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068835809999999997 2024-09-16T04:09:04.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520991+0000", 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6646999999999998e-05 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.546600+0000", 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025608881999999999 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:850 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~4063232,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.513985+0000", 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.591837291, 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035758587000000001, 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 850 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513985+0000", 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513985+0000", 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513986+0000", 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1739999999999999e-06 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514004+0000", 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7393000000000001e-05 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514014+0000", 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8859999999999999e-06 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514021+0000", 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6580000000000002e-06 2024-09-16T04:09:04.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514084+0000", 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2485999999999998e-05 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514105+0000", 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1185000000000001e-05 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.524619+0000", 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010513504999999999 2024-09-16T04:09:04.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.524728+0000", 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109324 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.549744+0000", 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025015975999999999 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.186 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:851 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~2146304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.514064+0000", 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.5917589219999999, 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013147891, 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 851 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514064+0000", 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514064+0000", 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514065+0000", 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.569999999999999e-07 2024-09-16T04:09:04.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514070+0000", 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8320000000000002e-06 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514078+0000", 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3939999999999998e-06 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514083+0000", 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8060000000000002e-06 2024-09-16T04:09:04.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514154+0000", 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1019000000000006e-05 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514169+0000", 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5112e-05 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.519800+0000", 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056317049999999999 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.519945+0000", 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000144155 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.527212+0000", 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007266911 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:852 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.514096+0000", 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.591727036, 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015213175000000001, 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 852 2024-09-16T04:09:04.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514096+0000", 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514096+0000", 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514096+0000", 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514109+0000", 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2979999999999999e-05 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514118+0000", 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8710000000000003e-06 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514124+0000", 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514211+0000", 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6284000000000003e-05 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514217+0000", 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0499999999999997e-06 2024-09-16T04:09:04.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520494+0000", 2024-09-16T04:09:04.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062772829999999998 2024-09-16T04:09:04.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520602+0000", 2024-09-16T04:09:04.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010760999999999999 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.529309+0000", 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0087071709999999997 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:853 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.514138+0000", 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.591684785, 2024-09-16T04:09:04.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017460188000000001, 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 853 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514138+0000", 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514138+0000", 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514138+0000", 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3900000000000004e-07 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514143+0000", 2024-09-16T04:09:04.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3409999999999996e-06 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514151+0000", 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0290000000000005e-06 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514157+0000", 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3990000000000001e-06 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514251+0000", 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3330000000000003e-05 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514256+0000", 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4240000000000001e-06 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.522521+0000", 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082649720000000006 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.522639+0000", 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000118114 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.531598+0000", 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089589400000000003 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpReadReply(3.4s0 86/56 ECSubReadReply(tid=46, attrs_read=0))", 2024-09-16T04:09:04.202 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.526934+0000", 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.578888732, 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015870366, 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.526934+0000", 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:04.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.526934+0000", 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.526935+0000", 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.536580+0000", 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0096454929999999998 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.536677+0000", 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7235e-05 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.536687+0000", 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8579999999999995e-06 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.536843+0000", 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015625799999999999 2024-09-16T04:09:04.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.542804+0000", 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059607979999999998 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:04.206 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:04.207 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:09:04.208 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:09:04.208 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:09:04.208 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:09:04.209 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:09:04.210 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:09:04.206+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:09:04.308 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:04.306+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:04.312 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:04.312 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:04.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:09:04.408 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:04.407+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:04.423 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:04.423 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:04.431 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:09:04.517 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:04.515+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:04.528 INFO:tasks.rados.rados.0.smithi016.stdout:361: done (1 left) 2024-09-16T04:09:04.528 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 25 current snap is 32 2024-09-16T04:09:04.528 INFO:tasks.rados.rados.0.smithi016.stdout:360: finishing write tid 1 to smithi01639990-33 2024-09-16T04:09:04.528 INFO:tasks.rados.rados.0.smithi016.stdout:362: seq_num 149 ranges {6209536=98304} 2024-09-16T04:09:04.564 INFO:tasks.rados.rados.0.smithi016.stdout:362: writing smithi01639990-25 from 6209536 to 6307840 tid 1 2024-09-16T04:09:04.564 INFO:tasks.rados.rados.0.smithi016.stdout:363: read oid 36 snap -1 2024-09-16T04:09:04.564 INFO:tasks.rados.rados.0.smithi016.stdout:363: expect deleted 2024-09-16T04:09:04.564 INFO:tasks.rados.rados.0.smithi016.stdout:364: read oid 27 snap 20 2024-09-16T04:09:04.565 INFO:tasks.rados.rados.0.smithi016.stdout:364: expect deleted 2024-09-16T04:09:04.565 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 42 current snap is 32 2024-09-16T04:09:04.565 INFO:tasks.rados.rados.0.smithi016.stdout:360: finishing write tid 2 to smithi01639990-33 2024-09-16T04:09:04.565 INFO:tasks.rados.rados.0.smithi016.stdout:360: finishing write tid 3 to smithi01639990-33 2024-09-16T04:09:04.565 INFO:tasks.rados.rados.0.smithi016.stdout:360: oid 33 updating version 0 to 62 2024-09-16T04:09:04.565 INFO:tasks.rados.rados.0.smithi016.stdout:360: oid 33 version 62 is already newer than 61 2024-09-16T04:09:04.565 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 33 v 62 (ObjNum 148 snap 32 seq_num 148) dirty exists 2024-09-16T04:09:04.565 INFO:tasks.rados.rados.0.smithi016.stdout:360: left oid 33 (ObjNum 148 snap 32 seq_num 148) 2024-09-16T04:09:04.565 INFO:tasks.rados.rados.0.smithi016.stdout:365: seq_num 150 ranges {3194880=114688} 2024-09-16T04:09:04.583 INFO:tasks.rados.rados.0.smithi016.stdout:365: writing smithi01639990-42 from 3194880 to 3309568 tid 1 2024-09-16T04:09:04.583 INFO:tasks.rados.rados.0.smithi016.stdout:360: done (4 left) 2024-09-16T04:09:04.583 INFO:tasks.rados.rados.0.smithi016.stdout:366: read oid 29 snap -1 2024-09-16T04:09:04.583 INFO:tasks.rados.rados.0.smithi016.stdout:366: expect deleted 2024-09-16T04:09:04.583 INFO:tasks.rados.rados.0.smithi016.stdout:367: read oid 15 snap -1 2024-09-16T04:09:04.583 INFO:tasks.rados.rados.0.smithi016.stdout:367: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-16T04:09:04.583 INFO:tasks.rados.rados.0.smithi016.stdout:368: rollback oid 16 current snap is 32 2024-09-16T04:09:04.584 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 16 to 20 2024-09-16T04:09:04.584 INFO:tasks.rados.rados.0.smithi016.stdout:362: finishing write tid 1 to smithi01639990-25 2024-09-16T04:09:04.584 INFO:tasks.rados.rados.0.smithi016.stdout:363: done (6 left) 2024-09-16T04:09:04.584 INFO:tasks.rados.rados.0.smithi016.stdout:364: done (5 left) 2024-09-16T04:09:04.584 INFO:tasks.rados.rados.0.smithi016.stdout:369: rollback oid 4 current snap is 32 2024-09-16T04:09:04.584 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 4 to 19 2024-09-16T04:09:04.584 INFO:tasks.rados.rados.0.smithi016.stdout:370: delete oid 17 current snap is 32 2024-09-16T04:09:04.585 INFO:tasks.rados.rados.0.smithi016.stdout:362: finishing write tid 2 to smithi01639990-25 2024-09-16T04:09:04.589 INFO:tasks.rados.rados.0.smithi016.stdout:365: finishing write tid 1 to smithi01639990-42 2024-09-16T04:09:04.589 INFO:tasks.rados.rados.0.smithi016.stdout:368: finishing rollback tid 0 to smithi01639990-16 2024-09-16T04:09:04.590 INFO:tasks.rados.rados.0.smithi016.stdout:368: finishing rollback tid 1 to smithi01639990-16 2024-09-16T04:09:04.593 INFO:tasks.rados.rados.0.smithi016.stdout:367: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-16T04:09:04.596 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:04.596 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:04.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:09:04.647 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:04.645+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:04.666 INFO:tasks.rados.rados.0.smithi016.stdout:365: finishing write tid 2 to smithi01639990-42 2024-09-16T04:09:04.666 INFO:tasks.rados.rados.0.smithi016.stdout:362: finishing write tid 3 to smithi01639990-25 2024-09-16T04:09:04.666 INFO:tasks.rados.rados.0.smithi016.stdout:362: oid 25 updating version 0 to 54 2024-09-16T04:09:04.667 INFO:tasks.rados.rados.0.smithi016.stdout:362: oid 25 version 54 is already newer than 53 2024-09-16T04:09:04.667 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 25 v 54 (ObjNum 149 snap 32 seq_num 149) dirty exists 2024-09-16T04:09:04.667 INFO:tasks.rados.rados.0.smithi016.stdout:362: left oid 25 (ObjNum 149 snap 32 seq_num 149) 2024-09-16T04:09:04.667 INFO:tasks.rados.rados.0.smithi016.stdout:369: finishing rollback tid 0 to smithi01639990-4 2024-09-16T04:09:04.700 INFO:tasks.rados.rados.0.smithi016.stdout:362: done (6 left) 2024-09-16T04:09:04.700 INFO:tasks.rados.rados.0.smithi016.stdout:366: done (5 left) 2024-09-16T04:09:04.700 INFO:tasks.rados.rados.0.smithi016.stdout:367: done (4 left) 2024-09-16T04:09:04.700 INFO:tasks.rados.rados.0.smithi016.stdout:370: done (3 left) 2024-09-16T04:09:04.700 INFO:tasks.rados.rados.0.smithi016.stdout:371: snap_create 2024-09-16T04:09:04.711 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:09:04.710+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:09:04.713 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:04.713 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:04.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:09:04.812 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:09:04.811+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:09:04.828 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:04.828 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:04.837 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:09:04.913 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:09:04.912+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:09:04.924 INFO:tasks.rados.rados.0.smithi016.stdout:369: finishing rollback tid 1 to smithi01639990-4 2024-09-16T04:09:04.944 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:04.944 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:04.951 INFO:tasks.rados.rados.0.smithi016.stdout:369: finishing rollback tid 2 to smithi01639990-4 2024-09-16T04:09:04.951 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 4 v 71 (ObjNum 273 snap 0 seq_num 0) dirty exists 2024-09-16T04:09:04.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:09:04.953 INFO:tasks.rados.rados.0.smithi016.stdout:365: finishing write tid 3 to smithi01639990-42 2024-09-16T04:09:04.954 INFO:tasks.rados.rados.0.smithi016.stdout:365: oid 42 updating version 0 to 66 2024-09-16T04:09:04.954 INFO:tasks.rados.rados.0.smithi016.stdout:365: oid 42 updating version 66 to 67 2024-09-16T04:09:04.954 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 42 v 67 (ObjNum 150 snap 32 seq_num 150) dirty exists 2024-09-16T04:09:04.954 INFO:tasks.rados.rados.0.smithi016.stdout:365: left oid 42 (ObjNum 150 snap 32 seq_num 150) 2024-09-16T04:09:05.014 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:09:05.013+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:09:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:05.061 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:05.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:09:05.083 INFO:tasks.rados.rados.0.smithi016.stdout:368: finishing rollback tid 2 to smithi01639990-16 2024-09-16T04:09:05.083 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 16 v 107 (ObjNum 57 snap 2 seq_num 57) dirty exists 2024-09-16T04:09:05.115 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:05.113+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:05.176 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:05.176 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:05.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:09:05.216 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:05.214+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:05.316 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:09:05.315+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:09:05.331 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:05.332 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:05.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:09:05.418 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:05.416+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:05.438 INFO:tasks.rados.rados.0.smithi016.stdout:365: done (3 left) 2024-09-16T04:09:05.438 INFO:tasks.rados.rados.0.smithi016.stdout:368: done (2 left) 2024-09-16T04:09:05.438 INFO:tasks.rados.rados.0.smithi016.stdout:369: done (1 left) 2024-09-16T04:09:05.439 INFO:tasks.rados.rados.0.smithi016.stdout:371: done (0 left) 2024-09-16T04:09:05.439 INFO:tasks.rados.rados.0.smithi016.stdout:372: snap_remove snap 31 2024-09-16T04:09:05.459 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-09-16T04:09:05.459 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:09:05.459 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-16T04:09:05.459 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T04:09:05.459 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T04:09:05.460 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-16T04:09:05.490 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:05.490 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:05.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:09:05.518 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:05.517+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:05.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:09:05.568 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:05.569 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T04:09:05.569 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:05.619 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:05.618+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:05.648 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:05.649 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:05.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:09:05.720 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:05.718+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:05.765 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:05.766 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:05.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:09:05.859 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:09:05.857+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:09:05.921 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:09:05.920+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:09:05.923 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:05.923 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:05.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:09:06.021 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:06.020+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:06.080 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:06.080 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:06.089 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:09:06.122 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:09:06.121+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:09:06.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:09:06.226 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:09:06.224+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:09:06.238 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:06.238 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:06.247 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:09:06.324 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:09:06.322+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:09:06.356 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:09:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:09:06.374 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:06.374 INFO:teuthology.orchestra.run.smithi174.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:06.378 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:09:06.378 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:09:06.383 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:09:06.398 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:09:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:09:06.419 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:09:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:09:06.424 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:09:06.423+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:09:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-16T04:09:06.439 INFO:tasks.rados.rados.0.smithi016.stdout:372: done (0 left) 2024-09-16T04:09:06.440 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 20 current snap is 33 2024-09-16T04:09:06.440 INFO:tasks.rados.rados.0.smithi016.stdout:373: seq_num 152 ranges {0=147456} 2024-09-16T04:09:06.440 INFO:tasks.rados.rados.0.smithi016.stdout:373: writing smithi01639990-20 from 0 to 147456 tid 1 2024-09-16T04:09:06.440 INFO:tasks.rados.rados.0.smithi016.stdout:374: snap_remove snap 20 2024-09-16T04:09:06.442 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:09:06.442 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:09:06.470 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:09:06.470 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:09:06.492 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:09:06.492 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:09:06.514 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:09:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:09:06.525 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:06.524+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:06.535 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:09:06.535 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:09:06.539 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:06.540 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:06.540 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:06.540 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:06.540 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:06.548 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:09:06.556 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:09:06.557 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:09:06.626 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:06.625+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:06.652 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:06.652 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:06.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:199 3.3s0 3:c05d4807:::smithi01639990-31:head [copy-from ver 11] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.439714367999997, 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.48371305599999997, 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:06.653 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 199 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209959+0000", 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7460000000000001e-06 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209971+0000", 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2787e-05 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210000+0000", 2024-09-16T04:09:06.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8952999999999999e-05 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210015+0000", 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.434e-05 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210169+0000", 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015455199999999999 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210317+0000", 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000147291 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.224854+0000", 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014536926 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226325+0000", 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001471099 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693612+0000", 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.467287324 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693669+0000", 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7037999999999998e-05 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:200 3.3s0 3:c05d4807:::smithi01639990-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.439624078999998, 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016139113, 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 200 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.659 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210047+0000", 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:09:06.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210051+0000", 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8800000000000001e-06 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210065+0000", 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3567000000000001e-05 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210074+0000", 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2399999999999996e-06 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210464+0000", 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00038978800000000002 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210496+0000", 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2539999999999997e-05 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.225987+0000", 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015491398999999999 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226084+0000", 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6618000000000002e-05 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226185+0000", 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101205 2024-09-16T04:09:06.663 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s6 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.550080139999999, 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010829642, 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.664 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099591+0000", 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4399999999999999e-07 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099619+0000", 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7555e-05 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099655+0000", 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6337000000000001e-05 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099691+0000", 2024-09-16T04:09:06.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5994000000000001e-05 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099819+0000", 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000128484 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099829+0000", 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1949999999999992e-06 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110366+0000", 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010537504 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110420+0000", 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3729e-05 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s2 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:09:06.668 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.548263507999998, 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012298224, 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000002e-07 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101422+0000", 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5004000000000001e-05 2024-09-16T04:09:06.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101446+0000", 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3632999999999999e-05 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101452+0000", 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0260000000000004e-06 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101511+0000", 2024-09-16T04:09:06.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9515000000000002e-05 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101523+0000", 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1094000000000001e-05 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113680+0000", 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012157401 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113705+0000", 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5041999999999998e-05 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.es3 33/27 ECSubWrite(tid=31, reqid=osd.11.0:31, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.548145157999997, 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012148897000000001, 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.673 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101526+0000", 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0900000000000001e-07 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101543+0000", 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7215999999999999e-05 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101584+0000", 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1352e-05 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101590+0000", 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5779999999999998e-06 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101650+0000", 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0093000000000002e-05 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101662+0000", 2024-09-16T04:09:06.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2257e-05 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113631+0000", 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.01196856 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113674+0000", 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3132e-05 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.548122141999997, 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012128856, 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:09:06.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2399999999999999e-07 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101558+0000", 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1980000000000007e-06 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101618+0000", 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0911e-05 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101624+0000", 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6899999999999997e-06 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101690+0000", 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5988000000000006e-05 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101701+0000", 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0878e-05 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113645+0000", 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011944110000000001 2024-09-16T04:09:06.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113677+0000", 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1856999999999997e-05 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.546643088000003, 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014214867000000001, 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.682 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.683 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103028+0000", 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8999999999999995e-07 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103051+0000", 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2909999999999999e-05 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103122+0000", 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1339000000000003e-05 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103130+0000", 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8790000000000003e-06 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104942+0000", 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001811893 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104952+0000", 2024-09-16T04:09:06.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5449999999999992e-06 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117201+0000", 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012249395 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117242+0000", 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1016000000000002e-05 2024-09-16T04:09:06.686 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 57.545115527999997, 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012207213999999999, 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104556+0000", 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:09:06.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104963+0000", 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000407604 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105023+0000", 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9684e-05 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105031+0000", 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3599999999999996e-06 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105239+0000", 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00020760900000000001 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105251+0000", 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1788e-05 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116731+0000", 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011480292 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116762+0000", 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1087000000000002e-05 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 51/48 ECSubWriteReply(tid=36, last_complete=51'22, committed=1, applied=1))", 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:26.486527+0000", 2024-09-16T04:09:06.691 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.163143276, 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.69968956299999996, 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486527+0000", 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486527+0000", 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486528+0000", 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5899999999999995e-07 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486532+0000", 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4449999999999998e-06 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.185640+0000", 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.69910766300000005 2024-09-16T04:09:06.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.185651+0000", 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1423999999999999e-05 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.185718+0000", 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6724999999999996e-05 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.186217+0000", 2024-09-16T04:09:06.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049854700000000001 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpRead(3.as1 57/56 ECSubRead(tid=39, to_read={3:5e52341f:::smithi01639990-38:d=0,262144,0}, subchunks={3:5e52341f:::smithi01639990-38:d=[0,1]}, attrs_to_read=))", 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.575872+0000", 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.073798296, 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.86215607299999997, 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.695 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.575872+0000", 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.575872+0000", 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.575873+0000", 2024-09-16T04:09:06.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5300000000000004e-07 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.575880+0000", 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4610000000000002e-06 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437261+0000", 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.86138053000000003 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437268+0000", 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6199999999999999e-06 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437339+0000", 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0621e-05 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.438028+0000", 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068918799999999998 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.698 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpRead(3.as1 58/56 ECSubRead(tid=40, to_read={3:5e52341f:::smithi01639990-38:head=0,262144,0}, subchunks={3:5e52341f:::smithi01639990-38:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:33.437230+0000", 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.212440690000001, 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.081032428000000004, 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437230+0000", 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437230+0000", 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437230+0000", 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6100000000000002e-07 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437239+0000", 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2690000000000005e-06 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.517631+0000", 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.080391539999999997 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.517655+0000", 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4771999999999999e-05 2024-09-16T04:09:06.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.517708+0000", 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2478999999999997e-05 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.518262+0000", 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055410699999999997 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.702 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:522 3.14s0 3:2d08ec72:::smithi01639990-14:head [copy-from ver 47] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:37.619670+0000", 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.030000622999999, 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.32033554400000003, 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 522 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619670+0000", 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619670+0000", 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619671+0000", 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5680000000000001e-06 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619677+0000", 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5160000000000002e-06 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619688+0000", 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1276e-05 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619694+0000", 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5899999999999998e-06 2024-09-16T04:09:06.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619749+0000", 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5640000000000003e-05 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619839+0000", 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9592000000000006e-05 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.643252+0000", 2024-09-16T04:09:06.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.023413481999999999 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.647039+0000", 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037869969999999998 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.939885+0000", 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.29284586099999999 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.940005+0000", 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000120022 2024-09-16T04:09:06.707 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:523 3.14s0 3:2d08ec72:::smithi01639990-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:37.619712+0000", 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.029958529000002, 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.026455427, 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 523 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619712+0000", 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619712+0000", 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619712+0000", 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8299999999999997e-07 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619717+0000", 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.707e-06 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619724+0000", 2024-09-16T04:09:06.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0260000000000002e-06 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619730+0000", 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6579999999999999e-06 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.620023+0000", 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00029293599999999999 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:06.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.620066+0000", 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3327e-05 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.646035+0000", 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.025969142000000001 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.646100+0000", 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4986000000000004e-05 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.646167+0000", 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7162000000000005e-05 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 62/48 ECSubWriteReply(tid=54, last_complete=62'37, committed=1, applied=1))", 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:37.649035+0000", 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.000635632000002, 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.29138913900000002, 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.713 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.649035+0000", 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.649035+0000", 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.649036+0000", 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.649041+0000", 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5060000000000003e-06 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.939461+0000", 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.290420077 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.939477+0000", 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5956000000000001e-05 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.939647+0000", 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00017009800000000001 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.940424+0000", 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077663899999999998 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.1s1 63/56 ECSubWrite(tid=110, reqid=osd.0.0:110, at_version=63'19, trim_to=0'0, roll_forward_to=62'17))", 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:38.513701+0000", 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.135969673000002, 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010654208, 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513701+0000", 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513701+0000", 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513701+0000", 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.35e-07 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513733+0000", 2024-09-16T04:09:06.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2359e-05 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513769+0000", 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5868999999999998e-05 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513783+0000", 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3995999999999999e-05 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514472+0000", 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068909500000000003 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514485+0000", 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2923e-05 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.524310+0000", 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098244760000000004 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.524355+0000", 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5255e-05 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.bs2 68/56 ECSubWrite(tid=324, reqid=client.4440.0:645, at_version=68'64, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.536111+0000", 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.113558821000002, 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098084000000000001, 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.536111+0000", 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.536111+0000", 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.536112+0000", 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3399999999999999e-07 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.536170+0000", 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8505999999999999e-05 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.536199+0000", 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8084000000000001e-05 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.536213+0000", 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4656e-05 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.536350+0000", 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000137057 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.536368+0000", 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7859999999999998e-05 2024-09-16T04:09:06.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545891+0000", 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095226779999999997 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545920+0000", 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9025000000000001e-05 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.bs2 68/56 ECSubWrite(tid=329, reqid=osd.3.0:329, at_version=68'68, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:09:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.541640+0000", 2024-09-16T04:09:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.108030674999998, 2024-09-16T04:09:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010394667999999999, 2024-09-16T04:09:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541640+0000", 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541640+0000", 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541640+0000", 2024-09-16T04:09:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.75e-07 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541669+0000", 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8578e-05 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541755+0000", 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5779999999999995e-05 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541764+0000", 2024-09-16T04:09:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3780000000000007e-06 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545213+0000", 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034488869999999999 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545223+0000", 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0581000000000001e-05 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.729 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551990+0000", 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067669560000000002 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.552034+0000", 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3933000000000001e-05 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.1s1 68/56 ECSubWrite(tid=134, reqid=osd.0.0:134, at_version=68'21, trim_to=0'0, roll_forward_to=63'19))", 2024-09-16T04:09:06.730 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.541687+0000", 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.107982968000002, 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010261344, 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541687+0000", 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541687+0000", 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541687+0000", 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.85e-07 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541731+0000", 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3461999999999998e-05 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545720+0000", 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0039894290000000001 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545729+0000", 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6330000000000002e-06 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545795+0000", 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5874e-05 2024-09-16T04:09:06.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545805+0000", 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0448000000000001e-05 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551878+0000", 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0060724719999999998 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551949+0000", 2024-09-16T04:09:06.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0840999999999995e-05 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.17s7 68/56 ECSubWrite(tid=144, reqid=client.4440.0:650, at_version=68'65, trim_to=0'0, roll_forward_to=68'61))", 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.722856+0000", 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 22.926814086, 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.052442728000000001, 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.722856+0000", 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.722856+0000", 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.722857+0000", 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1200000000000003e-07 2024-09-16T04:09:06.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.722871+0000", 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4571e-05 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.774611+0000", 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.051739988000000001 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.774617+0000", 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.774665+0000", 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7555999999999997e-05 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.774671+0000", 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5809999999999996e-06 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.775270+0000", 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059979300000000005 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.775299+0000", 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8547e-05 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.739 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.19s2 87/56 ECSubWrite(tid=191, reqid=client.4440.0:862, at_version=87'51, trim_to=0'0, roll_forward_to=67'48))", 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:03.021970+0000", 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.6277003529999998, 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.097252570999999996, 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021970+0000", 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021970+0000", 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021971+0000", 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2999999999999999e-07 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021999+0000", 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7888000000000001e-05 2024-09-16T04:09:06.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.118478+0000", 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.096479121000000001 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.118484+0000", 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7089999999999999e-06 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:06.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.118565+0000", 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1089999999999998e-05 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.118576+0000", 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1489e-05 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119194+0000", 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00061798300000000001 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:06.743 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119222+0000", 2024-09-16T04:09:06.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8360999999999998e-05 2024-09-16T04:09:06.744 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.744 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.744 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.744 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:06.744 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:06.744 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:06.745 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:09:06.746 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:06.744+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:06.780 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:09:06.780 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:09:06.801 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:09:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:09:06.846 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:06.844+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:06.889 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:06.889 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:06.889 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:09:06.889 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:06.897 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:09:06.947 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:09:06.945+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:09:07.041 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:07.041 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:07.042 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:07.042 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:07.042 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:07.048 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:07.046+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:07.050 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:09:07.149 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:07.148+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:07.154 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:07.154 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:07.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:07.154 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:07.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.154 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.048514998999998, 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013925085, 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103015+0000", 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103016+0000", 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5499999999999999e-07 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103038+0000", 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2416e-05 2024-09-16T04:09:07.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103157+0000", 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000118439 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103166+0000", 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9819999999999997e-06 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103225+0000", 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8996000000000002e-05 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103235+0000", 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9909999999999997e-06 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116899+0000", 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013664549999999999 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116941+0000", 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1455999999999997e-05 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s5 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 58.048489697999997, 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013929731000000001, 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.159 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103041+0000", 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5899999999999998e-07 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103057+0000", 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5778e-05 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103193+0000", 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0001359 2024-09-16T04:09:07.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103199+0000", 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1279999999999996e-06 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107717+0000", 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045183769999999996 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.107726+0000", 2024-09-16T04:09:07.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5420000000000005e-06 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116950+0000", 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092240229999999996 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116970+0000", 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0724e-05 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/27 ECSubWriteReply(tid=88, last_complete=42'39, committed=1, applied=1))", 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.703084465000003, 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49630058100000002, 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448447+0000", 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448451+0000", 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3030000000000001e-06 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944646+0000", 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49619510999999999 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944655+0000", 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8480000000000007e-06 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944717+0000", 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2065999999999998e-05 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944747+0000", 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9505999999999998e-05 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDScrubReserve(3.0s7,QREQUEST,e:47,from:11(0),reservation_nonce:4)", 2024-09-16T04:09:07.167 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.124362650999998, 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17961150300000001, 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027169+0000", 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027173+0000", 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1300000000000003e-06 2024-09-16T04:09:07.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206636+0000", 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17946284700000001 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206644+0000", 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9650000000000008e-06 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206712+0000", 2024-09-16T04:09:07.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8215999999999999e-05 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206735+0000", 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2789000000000002e-05 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206779+0000", 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4669000000000002e-05 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:414 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~770048] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.666988353999997, 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012099706, 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 414 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.172 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484542+0000", 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.484544+0000", 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7129999999999999e-06 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485808+0000", 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012637099999999999 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485825+0000", 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7722000000000001e-05 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485834+0000", 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9749999999999996e-06 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.485905+0000", 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1131999999999996e-05 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.486012+0000", 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000106214 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.487551+0000", 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015389830000000001 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.489803+0000", 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0022524839999999999 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496554+0000", 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067513879999999997 2024-09-16T04:09:07.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.496642+0000", 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7385000000000004e-05 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:416 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~294912 in=294912b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.660907942999998, 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16418209, 2024-09-16T04:09:07.177 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 416 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490624+0000", 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3069999999999999e-06 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490921+0000", 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00029678400000000002 2024-09-16T04:09:07.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490932+0000", 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1239e-05 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490938+0000", 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0379999999999996e-06 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490980+0000", 2024-09-16T04:09:07.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2401999999999998e-05 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491037+0000", 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6768000000000003e-05 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491591+0000", 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055345000000000004 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492749+0000", 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0011580970000000001 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654745+0000", 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16199656800000001 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654805+0000", 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9437e-05 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.182 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:417 3.cs0 3:38c68ed3:::smithi01639990-42:head [setxattr _header (57) in=64b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.660577924000002, 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67294876800000003, 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.183 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 417 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2699999999999999e-07 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490956+0000", 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0130000000000002e-06 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490961+0000", 2024-09-16T04:09:07.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1510000000000003e-06 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490966+0000", 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7960000000000003e-06 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491934+0000", 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00096816399999999998 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492020+0000", 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5827000000000003e-05 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492479+0000", 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045912299999999997 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492973+0000", 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049418400000000005 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:07.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163802+0000", 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67082863599999998 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163901+0000", 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9246999999999994e-05 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:418 3.cs0 3:38c68ed3:::smithi01639990-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:07.188 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.660556745000001, 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67426341400000001, 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 418 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8000000000000001e-07 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490976+0000", 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.973e-06 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490983+0000", 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4699999999999999e-06 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490988+0000", 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5509999999999999e-06 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493081+0000", 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020931029999999998 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493165+0000", 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3988999999999998e-05 2024-09-16T04:09:07.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164330+0000", 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67116498999999996 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164423+0000", 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000004e-05 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.165237+0000", 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081446800000000005 2024-09-16T04:09:07.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 52/27 ECSubWriteReply(tid=99, last_complete=52'26, committed=1, applied=1))", 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.658615054000002, 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16189895200000001, 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492916+0000", 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1699999999999998e-07 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492919+0000", 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6970000000000002e-06 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654619+0000", 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16170007 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654628+0000", 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9080000000000001e-06 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654685+0000", 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7349999999999998e-05 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654814+0000", 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012941 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 52/27 ECSubWriteReply(tid=100, last_complete=52'27, committed=1, applied=1))", 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.658172981999996, 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67093597800000004, 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493358+0000", 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493360+0000", 2024-09-16T04:09:07.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2019999999999998e-06 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163618+0000", 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67025769000000002 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163644+0000", 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6089999999999999e-05 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163713+0000", 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9326999999999995e-05 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164293+0000", 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00058030099999999997 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.201 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 63/56 ECSubWrite(tid=277, reqid=osd.3.0:277, at_version=63'79, trim_to=0'0, roll_forward_to=62'77))", 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.640183540999999, 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.70853546499999998, 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4099999999999999e-07 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511375+0000", 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7755000000000001e-05 2024-09-16T04:09:07.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219030+0000", 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.70765444700000002 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219046+0000", 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6152999999999999e-05 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219182+0000", 2024-09-16T04:09:07.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013656599999999999 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219190+0000", 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1079999999999993e-06 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219848+0000", 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065702000000000002 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219882+0000", 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4875000000000001e-05 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:537 3.19s0 3:9fd10cc9:::smithi01639990-50:head [append 0~622592] snapc 13=[13,12,10] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:39.519316+0000", 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 27.632214179999998, 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016449781, 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.206 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 537 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.519316+0000", 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.519316+0000", 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.519319+0000", 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2639999999999998e-06 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520728+0000", 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0014093459999999999 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520743+0000", 2024-09-16T04:09:07.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4647999999999999e-05 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520759+0000", 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6201999999999998e-05 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520815+0000", 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6076999999999999e-05 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.520918+0000", 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000103041 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.522275+0000", 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013567500000000001 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.525027+0000", 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0027523780000000002 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.535715+0000", 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010687665000000001 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.535766+0000", 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1409999999999997e-05 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.211 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:538 3.19s0 3:9fd10cc9:::smithi01639990-50:head [append 0~442368] snapc 13=[13,12,10] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:39.522591+0000", 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 27.628939035999998, 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013308851999999999, 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 538 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.522591+0000", 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.522591+0000", 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.522599+0000", 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.571e-06 2024-09-16T04:09:07.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523302+0000", 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00070285300000000003 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523316+0000", 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3647e-05 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523342+0000", 2024-09-16T04:09:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6285e-05 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523403+0000", 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1705000000000006e-05 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.523505+0000", 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010149099999999999 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.524458+0000", 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095253700000000005 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.526152+0000", 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001694152 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.535852+0000", 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097005860000000006 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.535900+0000", 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8025000000000002e-05 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 67/56 ECSubWriteReply(tid=146, last_complete=67'24, committed=1, applied=1))", 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:42.548179+0000", 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.603351235000002, 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.18309741900000001, 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.217 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548179+0000", 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548179+0000", 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548180+0000", 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4199999999999995e-07 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548184+0000", 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4700000000000004e-06 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.730637+0000", 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.182453118 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.730647+0000", 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0650000000000005e-06 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.730717+0000", 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0551999999999998e-05 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.731277+0000", 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055957200000000004 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.220 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds1 81/56 ECSubWrite(tid=221, reqid=osd.7.0:221, at_version=81'36, trim_to=0'0, roll_forward_to=80'34))", 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:56.485832+0000", 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.665698245, 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67555770299999995, 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485832+0000", 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485832+0000", 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485833+0000", 2024-09-16T04:09:07.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485854+0000", 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1855999999999999e-05 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160608+0000", 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67475406599999999 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160614+0000", 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.789e-06 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160698+0000", 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3311000000000002e-05 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160705+0000", 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.161360+0000", 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065466099999999996 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.161390+0000", 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0108999999999999e-05 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:780 3.cs0 3:38c68ed3:::smithi01639990-42:head [copy-from ver 22] snapc 1d=[1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.488449+0000", 2024-09-16T04:09:07.225 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.6630819300000006, 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.022970059000000001, 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 780 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488449+0000", 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488449+0000", 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488451+0000", 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7609999999999999e-06 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488464+0000", 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2547e-05 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488485+0000", 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0789999999999999e-05 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488497+0000", 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919000000000001e-05 2024-09-16T04:09:07.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488579+0000", 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2398999999999995e-05 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488588+0000", 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7309999999999999e-06 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490560+0000", 2024-09-16T04:09:07.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019719410000000001 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490628+0000", 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8275000000000006e-05 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.504399+0000", 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013771097 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506432+0000", 2024-09-16T04:09:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020333460000000001 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.511365+0000", 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049331410000000003 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.511419+0000", 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3112000000000002e-05 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.231 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:781 3.cs0 3:38c68ed3:::smithi01639990-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.488524+0000", 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.6630063279999998, 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017905164000000001, 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.232 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 781 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488524+0000", 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488524+0000", 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488525+0000", 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6400000000000002e-07 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488528+0000", 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2440000000000002e-06 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488537+0000", 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6209999999999993e-06 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488542+0000", 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6890000000000001e-06 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488657+0000", 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011471799999999999 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488663+0000", 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1330000000000004e-06 2024-09-16T04:09:07.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490738+0000", 2024-09-16T04:09:07.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020745709999999999 2024-09-16T04:09:07.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:07.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490765+0000", 2024-09-16T04:09:07.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6897999999999999e-05 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506037+0000", 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015272819 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506239+0000", 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00020141000000000001 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506429+0000", 2024-09-16T04:09:07.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000190497 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:817 3.cs0 3:338c6e39:::smithi01639990-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:58.504509+0000", 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 8.647020994, 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011997270000000001, 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:07.238 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 817 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504509+0000", 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504509+0000", 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504510+0000", 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8799999999999998e-07 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504511+0000", 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.361e-06 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504517+0000", 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1099999999999999e-06 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.504521+0000", 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7799999999999998e-06 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.507640+0000", 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00311883 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.507728+0000", 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7991999999999994e-05 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.515416+0000", 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0076882610000000001 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.515509+0000", 2024-09-16T04:09:07.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3313000000000006e-05 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:58.516507+0000", 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00099733500000000002 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:862 3.19s0 3:9fd10cc9:::smithi01639990-50:head [setxattr _header (57) in=7b] snapc 20=[1f,1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:03.019734+0000", 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.1317964260000002, 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099704702000000006, 2024-09-16T04:09:07.243 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 862 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019734+0000", 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019734+0000", 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019735+0000", 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019755+0000", 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0129999999999999e-05 2024-09-16T04:09:07.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019761+0000", 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6680000000000004e-06 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019767+0000", 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3299999999999998e-06 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021558+0000", 2024-09-16T04:09:07.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001791656 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021628+0000", 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9875000000000004e-05 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021980+0000", 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035168999999999999 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.022837+0000", 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00085693300000000004 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119380+0000", 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.096543290000000004 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119439+0000", 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8539999999999999e-05 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.248 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:863 3.19s0 3:9fd10cc9:::smithi01639990-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:03.019795+0000", 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.1317358500000001, 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.10150271399999999, 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 863 2024-09-16T04:09:07.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019795+0000", 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019795+0000", 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019795+0000", 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5499999999999998e-07 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019799+0000", 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6940000000000001e-06 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019806+0000", 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6650000000000003e-06 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019813+0000", 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9369999999999998e-06 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.022297+0000", 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024831499999999999 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.022386+0000", 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9072000000000004e-05 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119960+0000", 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.097574004000000006 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.120038+0000", 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8032000000000001e-05 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.121297+0000", 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012597050000000001 2024-09-16T04:09:07.253 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.254 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.254 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.254 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.254 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.254 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:07.255 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:09:07.256 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:07.253+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:07.356 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:07.354+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:07.359 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:07.359 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:07.359 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:09:07.359 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:07.368 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:09:07.441 INFO:tasks.rados.rados.0.smithi016.stdout:374: done (1 left) 2024-09-16T04:09:07.442 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 19 current snap is 33 2024-09-16T04:09:07.442 INFO:tasks.rados.rados.0.smithi016.stdout:373: finishing write tid 1 to smithi01639990-20 2024-09-16T04:09:07.442 INFO:tasks.rados.rados.0.smithi016.stdout:375: seq_num 153 ranges {3194880=163840} 2024-09-16T04:09:07.459 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:09:07.456+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:09:07.460 INFO:tasks.rados.rados.0.smithi016.stdout:375: writing smithi01639990-19 from 3194880 to 3358720 tid 1 2024-09-16T04:09:07.460 INFO:tasks.rados.rados.0.smithi016.stdout:376: read oid 46 snap -1 2024-09-16T04:09:07.460 INFO:tasks.rados.rados.0.smithi016.stdout:376: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:373: finishing write tid 2 to smithi01639990-20 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:373: finishing write tid 3 to smithi01639990-20 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:373: oid 20 updating version 0 to 73 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:373: oid 20 version 73 is already newer than 72 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 20 v 73 (ObjNum 152 snap 33 seq_num 152) dirty exists 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:373: left oid 20 (ObjNum 152 snap 33 seq_num 152) 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:373: done (2 left) 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:377: rmattr oid 33 current snap is 33 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:377: done (2 left) 2024-09-16T04:09:07.461 INFO:tasks.rados.rados.0.smithi016.stdout:378: delete oid 48 current snap is 33 2024-09-16T04:09:07.471 INFO:tasks.rados.rados.0.smithi016.stdout:375: finishing write tid 1 to smithi01639990-19 2024-09-16T04:09:07.477 INFO:tasks.rados.rados.0.smithi016.stdout:378: done (2 left) 2024-09-16T04:09:07.477 INFO:tasks.rados.rados.0.smithi016.stdout:379: snap_remove snap 29 2024-09-16T04:09:07.531 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:07.532 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:07.532 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:07.541 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:09:07.559 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:09:07.557+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:09:07.659 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:07.658+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:07.685 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:07.685 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:07.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:07.685 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 50.23272901, 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091491550000000008, 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5800000000000001e-07 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450512+0000", 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057160000000000002 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450541+0000", 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8835000000000001e-05 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450549+0000", 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3080000000000008e-06 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450607+0000", 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7763999999999998e-05 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450616+0000", 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459045+0000", 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084290709999999998 2024-09-16T04:09:07.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459089+0000", 2024-09-16T04:09:07.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4233000000000002e-05 2024-09-16T04:09:07.690 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.690 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:469 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~491520] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.129896045999999, 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098323030000000006, 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.691 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 469 2024-09-16T04:09:07.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.692 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:09:07.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552774+0000", 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1999999999999999e-06 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553284+0000", 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00050997900000000003 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553297+0000", 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2723e-05 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553306+0000", 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5969999999999992e-06 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.694 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553363+0000", 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6922999999999997e-05 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553436+0000", 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3090000000000007e-05 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.554087+0000", 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065173499999999997 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.695 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.558220+0000", 2024-09-16T04:09:07.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041326330000000001 2024-09-16T04:09:07.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.696 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.696 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:07.696 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562517+0000", 2024-09-16T04:09:07.696 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042965859999999998 2024-09-16T04:09:07.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562605+0000", 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8837000000000003e-05 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:474 3.as0 3:5e52341f:::smithi01639990-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:09:07.697 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.122564085, 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097686969999999998, 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 474 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:09:07.698 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560106+0000", 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.63e-07 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.699 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560109+0000", 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.303e-06 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560116+0000", 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3860000000000001e-06 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560122+0000", 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4450000000000004e-06 2024-09-16T04:09:07.700 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.701 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.701 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563686+0000", 2024-09-16T04:09:07.701 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035639740000000001 2024-09-16T04:09:07.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.701 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.702 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:07.702 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563776+0000", 2024-09-16T04:09:07.702 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0278000000000005e-05 2024-09-16T04:09:07.702 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568854+0000", 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0050781680000000001 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568970+0000", 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011579899999999999 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.703 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.569874+0000", 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00090388100000000004 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.244932390000002, 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015912599999999999, 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.704 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.705 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437757+0000", 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9735e-05 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437829+0000", 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1605000000000003e-05 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.706 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437838+0000", 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442352+0000", 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045136799999999999 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442364+0000", 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2642e-05 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.707 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453618+0000", 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011254128 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453649+0000", 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1016000000000003e-05 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.708 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.243893780999997, 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014897297, 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.709 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438776+0000", 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3099999999999997e-07 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438797+0000", 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0596000000000001e-05 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.710 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438836+0000", 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9684000000000002e-05 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438845+0000", 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6479999999999992e-06 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442677+0000", 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038317239999999999 2024-09-16T04:09:07.711 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442685+0000", 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453654+0000", 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010968637 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453673+0000", 2024-09-16T04:09:07.712 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8967999999999999e-05 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.241903831999998, 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012700487999999999, 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.713 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8900000000000001e-07 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.714 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440785+0000", 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9850000000000001e-05 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445485+0000", 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046992329999999997 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445495+0000", 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0506000000000001e-05 2024-09-16T04:09:07.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445668+0000", 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000172905 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445679+0000", 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453438+0000", 2024-09-16T04:09:07.716 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077588930000000002 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453466+0000", 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8175e-05 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=110, reqid=osd.5.0:109, at_version=0'0, trim_to=0'0, roll_forward_to=57'31))", 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.218099379999998, 2024-09-16T04:09:07.717 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011518758, 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.718 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8e-07 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464577+0000", 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9940000000000003e-06 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472531+0000", 2024-09-16T04:09:07.719 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079544330000000003 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472538+0000", 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7789999999999998e-06 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472599+0000", 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0841999999999998e-05 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.720 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472610+0000", 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1185999999999999e-05 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476060+0000", 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034496650000000002 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476089+0000", 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8679000000000001e-05 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.721 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=37, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.194596578000002, 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.039596059000000003, 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.722 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1e-07 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488075+0000", 2024-09-16T04:09:07.723 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8789999999999999e-06 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526262+0000", 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.038187556999999997 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526270+0000", 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3209999999999999e-06 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526424+0000", 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015469 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.527669+0000", 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012444019999999999 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=38, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.152104301000001, 2024-09-16T04:09:07.725 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028484058, 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.726 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530566+0000", 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.85e-07 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530570+0000", 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7319999999999998e-06 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558866+0000", 2024-09-16T04:09:07.727 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028295515 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558875+0000", 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7849999999999997e-06 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558966+0000", 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1611000000000001e-05 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.559049+0000", 2024-09-16T04:09:07.728 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.263e-05 2024-09-16T04:09:07.729 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.729 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.729 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.729 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.729 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.729 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 69/27 ECSubWrite(tid=153, reqid=osd.5.0:153, at_version=69'41, trim_to=0'0, roll_forward_to=68'39))", 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.143350883, 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0089824069999999995, 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.730 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539343+0000", 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4399000000000001e-05 2024-09-16T04:09:07.731 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539420+0000", 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7167000000000002e-05 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539428+0000", 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0679999999999997e-06 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539506+0000", 2024-09-16T04:09:07.732 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7836000000000005e-05 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539518+0000", 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1627000000000001e-05 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548257+0000", 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087393379999999993 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.733 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548301+0000", 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3798999999999999e-05 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.14s5 69/48 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=69'49, trim_to=0'0, roll_forward_to=68'45))", 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.143306633000002, 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0093116210000000008, 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.734 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0999999999999998e-07 2024-09-16T04:09:07.735 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539388+0000", 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4769999999999998e-05 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539439+0000", 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1149000000000001e-05 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539450+0000", 2024-09-16T04:09:07.736 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0557000000000001e-05 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539608+0000", 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000158679 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539620+0000", 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1260000000000001e-05 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.737 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548634+0000", 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009014256 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548674+0000", 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0240000000000001e-05 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.16s7 69/56 ECSubWrite(tid=152, reqid=osd.5.0:152, at_version=69'28, trim_to=0'0, roll_forward_to=68'26))", 2024-09-16T04:09:07.738 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.539883+0000", 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.142786507, 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085560289999999997, 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539883+0000", 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.739 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539883+0000", 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539884+0000", 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189999999999999e-06 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539909+0000", 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5621000000000001e-05 2024-09-16T04:09:07.740 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539956+0000", 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7209999999999997e-05 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539967+0000", 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0514e-05 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540036+0000", 2024-09-16T04:09:07.741 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9221999999999996e-05 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540047+0000", 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0898e-05 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548370+0000", 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0083228469999999995 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548439+0000", 2024-09-16T04:09:07.742 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8697999999999999e-05 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds3 69/56 ECSubWrite(tid=201, reqid=client.4440.0:654, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.142104194000002, 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098306729999999998, 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.743 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6600000000000002e-07 2024-09-16T04:09:07.744 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540718+0000", 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015259 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540851+0000", 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013328999999999999 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.745 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540860+0000", 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7909999999999993e-06 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541117+0000", 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025658799999999998 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541127+0000", 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0834e-05 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.550371+0000", 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092436089999999999 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.550396+0000", 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4505e-05 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.12s6 69/56 ECSubWrite(tid=147, reqid=osd.0.0:147, at_version=69'84, trim_to=0'0, roll_forward_to=68'82))", 2024-09-16T04:09:07.747 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.542608+0000", 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.140060722000001, 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021420679000000001, 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542608+0000", 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.748 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542608+0000", 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542609+0000", 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.15e-07 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542633+0000", 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4542999999999999e-05 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.749 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542748+0000", 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000114911 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542756+0000", 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542862+0000", 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000105689 2024-09-16T04:09:07.750 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542873+0000", 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0575e-05 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.563997+0000", 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021124446000000002 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564029+0000", 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2101000000000003e-05 2024-09-16T04:09:07.751 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.12s6 69/56 ECSubWrite(tid=149, reqid=osd.0.0:149, at_version=69'86, trim_to=0'0, roll_forward_to=68'82))", 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.544392+0000", 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.138277128999999, 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019668749999999999, 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544392+0000", 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544392+0000", 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544393+0000", 2024-09-16T04:09:07.753 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.195e-06 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544420+0000", 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6635000000000002e-05 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544463+0000", 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3479000000000002e-05 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.754 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544472+0000", 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8880000000000003e-06 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.551267+0000", 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00679526 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.551277+0000", 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2760000000000006e-06 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.755 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564037+0000", 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012760258 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564061+0000", 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3759000000000001e-05 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.756 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds3 69/56 ECSubWrite(tid=203, reqid=osd.7.0:203, at_version=69'28, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.546802+0000", 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.135867027, 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0088412939999999995, 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546802+0000", 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546802+0000", 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546802+0000", 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.96e-07 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546839+0000", 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6677999999999997e-05 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.758 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546944+0000", 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010538400000000001 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.546951+0000", 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3389999999999998e-06 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.547280+0000", 2024-09-16T04:09:07.759 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00032936 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.547291+0000", 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1007e-05 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.555601+0000", 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0083102599999999999 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.760 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.555643+0000", 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.197e-05 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 69/56 ECSubWrite(tid=340, reqid=osd.3.0:340, at_version=69'98, trim_to=0'0, roll_forward_to=68'96))", 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.128260401999999, 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096797529999999993, 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.761 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.762 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554437+0000", 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7382e-05 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554469+0000", 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2073000000000003e-05 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.763 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554480+0000", 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1358999999999999e-05 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554550+0000", 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9808999999999995e-05 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554564+0000", 2024-09-16T04:09:07.764 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3634e-05 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564066+0000", 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095028419999999992 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564088+0000", 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1996000000000001e-05 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.765 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 74/56 ECSubWrite(tid=376, reqid=osd.3.0:376, at_version=74'104, trim_to=0'0, roll_forward_to=70'102))", 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:49.453728+0000", 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 18.228940648999998, 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092035429999999998, 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.766 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453728+0000", 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453728+0000", 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453729+0000", 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1e-07 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.767 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453787+0000", 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8541000000000001e-05 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453868+0000", 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1199000000000006e-05 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453875+0000", 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8310000000000003e-06 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.768 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453979+0000", 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000103288 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453988+0000", 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.533e-06 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.462890+0000", 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0089015139999999993 2024-09-16T04:09:07.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.462932+0000", 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2426999999999999e-05 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:736 3.as0 3:5e52341f:::smithi01639990-38:head [copy-from ver 69] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439036+0000", 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.243633167, 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.030166564999999999, 2024-09-16T04:09:07.770 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 736 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439036+0000", 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.771 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439036+0000", 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439038+0000", 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4339999999999999e-06 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439050+0000", 2024-09-16T04:09:07.772 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1418e-05 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439070+0000", 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0251e-05 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439080+0000", 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4599999999999992e-06 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.773 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439154+0000", 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4820999999999994e-05 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439162+0000", 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6969999999999993e-06 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.443704+0000", 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045422580000000004 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.774 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.443816+0000", 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011184900000000001 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.461800+0000", 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017984236000000001 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.465505+0000", 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037041209999999999 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.775 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.469122+0000", 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036175270000000002 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.469203+0000", 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0493000000000003e-05 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:737 3.as0 3:5e52341f:::smithi01639990-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:07.776 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439252+0000", 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.243416987, 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.024601304000000001, 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 737 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:07.777 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439252+0000", 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439252+0000", 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439253+0000", 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7899999999999998e-07 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.778 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439259+0000", 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3609999999999997e-06 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439269+0000", 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8430000000000005e-06 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439276+0000", 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4649999999999997e-06 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.779 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439328+0000", 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1063999999999998e-05 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439334+0000", 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.443976+0000", 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046418049999999997 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.780 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.444023+0000", 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6872000000000001e-05 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.463639+0000", 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019616097999999998 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.463746+0000", 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010683600000000001 2024-09-16T04:09:07.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.463853+0000", 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010751600000000001 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:07.782 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:07.784 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:09:07.784 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:09:07.782+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:09:07.884 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:07.883+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:07.927 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:07.927 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:07.927 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:09:07.928 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:07.936 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:09:07.985 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:07.984+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:08.038 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:08.038 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:08.038 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:08.038 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:08.038 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:08.046 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:09:08.086 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:08.085+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:08.150 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 78.206798672999994, 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.151 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:09:08.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:09:08.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:09:08.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:08.158 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 77.224394430000004, 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:09:08.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 77.214031923999997, 2024-09-16T04:09:08.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:09:08.165 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.165 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.165 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:08.165 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:08.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:09:08.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:09:08.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:09:08.169 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.169 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.169 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.169 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.717443872000004, 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:09:08.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:09:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:165 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 75.262815317999994, 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011872888999999999, 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 165 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884706+0000", 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884707+0000", 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9900000000000002e-07 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884710+0000", 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3589999999999999e-06 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884714+0000", 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5460000000000002e-06 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.884717+0000", 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.021e-06 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894159+0000", 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0094419770000000007 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894201+0000", 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2074999999999999e-05 2024-09-16T04:09:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.894269+0000", 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7959000000000004e-05 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.895064+0000", 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00079458299999999996 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896521+0000", 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001457449 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896547+0000", 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.531e-05 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.896579+0000", 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2311000000000003e-05 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 61.038546779999997, 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:09:08.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 60.460612820000001, 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:09:08.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:09:08.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 59.043799675000002, 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.190 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:08.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 59.042412261000003, 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:09:08.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.683093231000001, 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.199 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:09:08.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:09:08.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.203 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.710223280999998, 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019153804, 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437299+0000", 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437320+0000", 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1121999999999999e-05 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437388+0000", 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7889e-05 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437397+0000", 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4390000000000006e-06 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441658+0000", 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042607110000000004 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441667+0000", 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:09:08.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456416+0000", 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014748493 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456452+0000", 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6159999999999999e-05 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 35.70703357, 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017174697999999999, 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440489+0000", 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.015e-06 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440509+0000", 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0338000000000001e-05 2024-09-16T04:09:08.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440547+0000", 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440557+0000", 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7820000000000005e-06 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442068+0000", 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015112179999999999 2024-09-16T04:09:08.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442078+0000", 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4620000000000002e-06 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457639+0000", 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015560954 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457663+0000", 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4121000000000002e-05 2024-09-16T04:09:08.212 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.as7 68/56 ECSubWrite(tid=57, reqid=osd.6.0:57, at_version=68'42, trim_to=0'0, roll_forward_to=64'40))", 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.613366318000001, 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017528866000000001, 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.213 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.49e-07 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534179+0000", 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3308000000000001e-05 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546651+0000", 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012471873 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546660+0000", 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2229999999999996e-06 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546730+0000", 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0123000000000006e-05 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546741+0000", 2024-09-16T04:09:08.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0684e-05 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551646+0000", 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049059180000000004 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551684+0000", 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7588e-05 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.217 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 68/56 ECSubWriteReply(tid=200, last_complete=68'24, committed=1, applied=1))", 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 24.584354294000001, 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46574167999999999, 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563168+0000", 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4099999999999997e-07 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563174+0000", 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.327e-06 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028683+0000", 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46550948399999997 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028698+0000", 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4576000000000001e-05 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028858+0000", 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016062100000000001 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028909+0000", 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0730999999999999e-05 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.221 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:654 3.ds0 3:bc502819:::smithi01639990-5:head [create,setxattr XXWdXV_kiPaajZX\\PgmqNaZqSVmab^hrtRYeagZr (1894) in=40b,setxattr NdZr`e\\^kXiksh]\\XocenQrhYc]c`[proOfo`Pd^ (33) in=40b,setxattr _`NqSdldlQbUNl^ZNRba\\naZhhdraiOgpafrZb[o (103) in=40b,setxattr iadimrq_[jOPUblhnVqTYqQNedg (1647) in=27b,setxattr `e\\rtQi]nmoP\\cm]oRYN`[]^qVUTsU^_oXm]h\\ec (6452) in=40b,setxattr ppZXc_dYi_WWXcUdeqcpck_ceUVa^XftgnfgdP[q (3576) in=40b,setxattr hehZsP\\ajqhe^nfrZjQmsnfWoqq\\e\\NV^rOaZYtj (1787) in=40b,setxattr a]Smbg`^k`emWabQRYcpr_lQj`eh`[r\\riqa_sfd (1015) in=40b,setxattr eqsaZkZ[`t^R^grhYOVrVsspVQQp_Zo\\loUV[\\U[ (982) in=40b,setxattr Rn`k`cSYQj`XOZdn_mSPrqQtoUpWTk[^[t (1647) in=34b,setxattr ]TtVabQgUTabfmqae[UVe`cNsqqYfdh]\\sVoeXQr (383) in=40b,setxattr iRn][N_NY\\ZhmYdQ[[ptTP_NW^Z]_roZnOpcmR_Y (19016) in=40b,setxattr T_b_WhnZnkaaYkNmmOUnaZa`dhRhQSshYQVQnVql (13472) in=40b,setxattr \\hZXh`dT^W[c\\XodZZ_d\\QO\\\\[PVWthO_WVOPSQd (1681) in=40b,setxattr ctSOqNY_dUqTOjgXYjqgslgNRRPoT\\\\OXPPWrtt_ (522) in=40b,setxattr f\\hTOf^sbRaYeUntSV^WhUo]qV`bSNermptfXs]Q (1645) in=40b,setxattr ohNTdfibmYs`eUTiQa^bcYW\\aarhT]YQ_haagqpf (1119) in=40b,setxattr qqUtZZ]ataUj_[pnh\\jVtU_dRpi^g[VPNeS`Y\\hk (699) in=40b,setxattr qsgedRfqdW]]cYbWOUgh^ZleaSQiteejr^afXbUj (525) in=40b,setxattr [XmPqpoSg`fWjdlmQ_qabof]_WVcPaY[]k[ardgj (594) in=40b] snapc 15=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.613343259000001, 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019213674, 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:08.222 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 654 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534181+0000", 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5830000000000001e-06 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534398+0000", 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000216663 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534421+0000", 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3714e-05 2024-09-16T04:09:08.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534434+0000", 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3185e-05 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534504+0000", 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9665000000000005e-05 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:08.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534510+0000", 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1709999999999999e-06 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.538902+0000", 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043914660000000001 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539044+0000", 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014215000000000001 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540400+0000", 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013559189999999999 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541692+0000", 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001291964 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553277+0000", 2024-09-16T04:09:08.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011585432999999999 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553392+0000", 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011476099999999999 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:733 3.fs0 3:f7af954f:::smithi01639990-10:head [append 0~0] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.70871161, 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016112728999999999, 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 733 2024-09-16T04:09:08.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.230 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438813+0000", 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6089999999999999e-06 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438822+0000", 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6199999999999994e-06 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438844+0000", 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1933000000000002e-05 2024-09-16T04:09:08.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438856+0000", 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1588e-05 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438966+0000", 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011003899999999999 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:08.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438974+0000", 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5019999999999992e-06 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.446497+0000", 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075231389999999999 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.446592+0000", 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4358999999999996e-05 2024-09-16T04:09:08.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.447539+0000", 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00094693900000000005 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454766+0000", 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0072270520000000003 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454838+0000", 2024-09-16T04:09:08.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1970000000000004e-05 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454923+0000", 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4979000000000001e-05 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:734 3.fs0 3:f7af954f:::smithi01639990-10:head [rollback 1a] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.708637556999999, 2024-09-16T04:09:08.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019165187, 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 734 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7399999999999998e-07 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438890+0000", 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8889999999999999e-06 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438903+0000", 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2466000000000001e-05 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438909+0000", 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3640000000000004e-06 2024-09-16T04:09:08.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439564+0000", 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000655084 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439571+0000", 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.472e-06 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448110+0000", 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085393550000000002 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448181+0000", 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1259000000000003e-05 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448725+0000", 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00054394000000000003 2024-09-16T04:09:08.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.457803+0000", 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0090773959999999997 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.457955+0000", 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015255800000000001 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458049+0000", 2024-09-16T04:09:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3930000000000004e-05 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:735 3.fs0 3:f7af954f:::smithi01639990-10:head [append 0~0] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.707935469000001, 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019107652999999999, 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:08.242 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 735 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:09:08.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439587+0000", 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439591+0000", 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2570000000000001e-06 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439607+0000", 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5846e-05 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439614+0000", 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6579999999999996e-06 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439657+0000", 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3402999999999998e-05 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.245 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439663+0000", 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0959999999999997e-06 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449536+0000", 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098724250000000006 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449600+0000", 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4857999999999995e-05 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.246 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449969+0000", 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036863400000000001 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458052+0000", 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0080833569999999993 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458113+0000", 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0764999999999998e-05 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.247 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458694+0000", 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00058057699999999998 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 81/27 ECSubWriteReply(tid=222, last_complete=81'16, committed=1, applied=1))", 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 11.649027933999999, 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.59824786600000002, 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.248 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:08.249 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9100000000000003e-07 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498500+0000", 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3290000000000001e-06 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096610+0000", 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.59810986399999999 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.250 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096617+0000", 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9609999999999999e-06 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096710+0000", 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3670000000000003e-05 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096741+0000", 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1251000000000003e-05 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.251 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 81/56 ECSubWriteReply(tid=223, last_complete=81'36, committed=1, applied=1))", 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.162644+0000", 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 10.984877285, 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.32306014599999999, 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162644+0000", 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162644+0000", 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162645+0000", 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3699999999999999e-07 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162658+0000", 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3112000000000001e-05 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.253 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485612+0000", 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.32295339699999998 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485618+0000", 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4489999999999996e-06 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485676+0000", 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7414999999999998e-05 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:08.254 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:08.255 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485704+0000", 2024-09-16T04:09:08.255 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8836e-05 2024-09-16T04:09:08.255 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.255 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.255 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.255 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:08.255 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:08.255 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:08.257 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:09:08.257 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:09:08.255+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:09:08.358 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:08.356+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:08.371 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:08.371 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:08.371 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:09:08.371 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:08.380 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:09:08.443 INFO:tasks.rados.rados.0.smithi016.stdout:379: done (2 left) 2024-09-16T04:09:08.443 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 20 current snap is 33 2024-09-16T04:09:08.443 INFO:tasks.rados.rados.0.smithi016.stdout:376: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T04:09:08.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:09:08.465 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:08.463+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:08.469 INFO:tasks.rados.rados.0.smithi016.stdout:375: finishing write tid 2 to smithi01639990-19 2024-09-16T04:09:08.469 INFO:tasks.rados.rados.0.smithi016.stdout:375: finishing write tid 3 to smithi01639990-19 2024-09-16T04:09:08.469 INFO:tasks.rados.rados.0.smithi016.stdout:375: oid 19 updating version 0 to 33 2024-09-16T04:09:08.469 INFO:tasks.rados.rados.0.smithi016.stdout:375: oid 19 version 33 is already newer than 32 2024-09-16T04:09:08.469 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 19 v 33 (ObjNum 153 snap 33 seq_num 153) dirty exists 2024-09-16T04:09:08.469 INFO:tasks.rados.rados.0.smithi016.stdout:375: left oid 19 (ObjNum 153 snap 33 seq_num 153) 2024-09-16T04:09:08.469 INFO:tasks.rados.rados.0.smithi016.stdout:380: seq_num 155 ranges {147456=212992} 2024-09-16T04:09:08.471 INFO:tasks.rados.rados.0.smithi016.stdout:380: writing smithi01639990-20 from 147456 to 360448 tid 1 2024-09-16T04:09:08.471 INFO:tasks.rados.rados.0.smithi016.stdout:375: done (2 left) 2024-09-16T04:09:08.471 INFO:tasks.rados.rados.0.smithi016.stdout:376: done (1 left) 2024-09-16T04:09:08.471 INFO:tasks.rados.rados.0.smithi016.stdout:381: delete oid 36 current snap is 33 2024-09-16T04:09:08.474 INFO:tasks.rados.rados.0.smithi016.stdout:381: done (1 left) 2024-09-16T04:09:08.474 INFO:tasks.rados.rados.0.smithi016.stdout:382: read oid 9 snap -1 2024-09-16T04:09:08.474 INFO:tasks.rados.rados.0.smithi016.stdout:382: expect (ObjNum 118 snap 20 seq_num 118) 2024-09-16T04:09:08.474 INFO:tasks.rados.rados.0.smithi016.stdout:383: setattr oid 4 current snap is 33 2024-09-16T04:09:08.476 INFO:tasks.rados.rados.0.smithi016.stdout:384: delete oid 1 current snap is 33 2024-09-16T04:09:08.480 INFO:tasks.rados.rados.0.smithi016.stdout:380: finishing write tid 1 to smithi01639990-20 2024-09-16T04:09:08.480 INFO:tasks.rados.rados.0.smithi016.stdout:380: finishing write tid 2 to smithi01639990-20 2024-09-16T04:09:08.482 INFO:tasks.rados.rados.0.smithi016.stdout:380: finishing write tid 3 to smithi01639990-20 2024-09-16T04:09:08.482 INFO:tasks.rados.rados.0.smithi016.stdout:380: oid 20 updating version 0 to 93 2024-09-16T04:09:08.482 INFO:tasks.rados.rados.0.smithi016.stdout:380: oid 20 version 93 is already newer than 92 2024-09-16T04:09:08.482 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 20 v 93 (ObjNum 155 snap 33 seq_num 155) dirty exists 2024-09-16T04:09:08.482 INFO:tasks.rados.rados.0.smithi016.stdout:380: left oid 20 (ObjNum 155 snap 33 seq_num 155) 2024-09-16T04:09:08.483 INFO:tasks.rados.rados.0.smithi016.stdout:380: done (3 left) 2024-09-16T04:09:08.483 INFO:tasks.rados.rados.0.smithi016.stdout:384: done (2 left) 2024-09-16T04:09:08.483 INFO:tasks.rados.rados.0.smithi016.stdout:385: rmattr oid 21 current snap is 33 2024-09-16T04:09:08.483 INFO:tasks.rados.rados.0.smithi016.stdout:385: done (2 left) 2024-09-16T04:09:08.483 INFO:tasks.rados.rados.0.smithi016.stdout:386: copy_from oid 2 from oid 17 current snap is 33 2024-09-16T04:09:08.483 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 4 v 95 (ObjNum 33 snap 0 seq_num 0) dirty exists 2024-09-16T04:09:08.484 INFO:tasks.rados.rados.0.smithi016.stdout:383: done (2 left) 2024-09-16T04:09:08.484 INFO:tasks.rados.rados.0.smithi016.stdout:387: copy_from oid 31 from oid 23 current snap is 33 2024-09-16T04:09:08.484 INFO:tasks.rados.rados.0.smithi016.stdout:388: read oid 21 snap -1 2024-09-16T04:09:08.485 INFO:tasks.rados.rados.0.smithi016.stdout:388: expect (ObjNum 125 snap 22 seq_num 125) 2024-09-16T04:09:08.485 INFO:tasks.rados.rados.0.smithi016.stdout:389: read oid 48 snap -1 2024-09-16T04:09:08.485 INFO:tasks.rados.rados.0.smithi016.stdout:389: expect deleted 2024-09-16T04:09:08.485 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 11 current snap is 33 2024-09-16T04:09:08.485 INFO:tasks.rados.rados.0.smithi016.stdout:390: seq_num 158 ranges {0=262144} 2024-09-16T04:09:08.485 INFO:tasks.rados.rados.0.smithi016.stdout:390: writing smithi01639990-11 from 0 to 262144 tid 1 2024-09-16T04:09:08.485 INFO:tasks.rados.rados.0.smithi016.stdout:391: delete oid 38 current snap is 33 2024-09-16T04:09:08.490 INFO:tasks.rados.rados.0.smithi016.stdout:386: finishing copy_from to smithi01639990-2 2024-09-16T04:09:08.490 INFO:tasks.rados.rados.0.smithi016.stdout:386: got expected ENOENT (src dne) 2024-09-16T04:09:08.490 INFO:tasks.rados.rados.0.smithi016.stdout:386: finishing copy_from racing read to smithi01639990-2 2024-09-16T04:09:08.494 INFO:tasks.rados.rados.0.smithi016.stdout:382: expect (ObjNum 118 snap 20 seq_num 118) 2024-09-16T04:09:08.546 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:08.546 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:08.546 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:09:08.546 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:09:08.546 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:08.555 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:09:08.597 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:08.595+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:08.631 INFO:tasks.rados.rados.0.smithi016.stdout:390: finishing write tid 1 to smithi01639990-11 2024-09-16T04:09:08.631 INFO:tasks.rados.rados.0.smithi016.stdout:390: finishing write tid 2 to smithi01639990-11 2024-09-16T04:09:08.631 INFO:tasks.rados.rados.0.smithi016.stdout:390: finishing write tid 3 to smithi01639990-11 2024-09-16T04:09:08.631 INFO:tasks.rados.rados.0.smithi016.stdout:390: oid 11 updating version 0 to 83 2024-09-16T04:09:08.631 INFO:tasks.rados.rados.0.smithi016.stdout:390: oid 11 version 83 is already newer than 82 2024-09-16T04:09:08.631 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 11 v 83 (ObjNum 158 snap 33 seq_num 158) dirty exists 2024-09-16T04:09:08.631 INFO:tasks.rados.rados.0.smithi016.stdout:390: left oid 11 (ObjNum 158 snap 33 seq_num 158) 2024-09-16T04:09:08.631 INFO:tasks.rados.rados.0.smithi016.stdout:388: expect (ObjNum 125 snap 22 seq_num 125) 2024-09-16T04:09:08.660 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:08.658+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:08.700 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:09:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:09:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:09:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.700 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 59.555936389000003, 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.701 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:09:08.703 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:09:08.704 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.237880781000001, 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.705 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:09:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:09:08.707 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:09:08.708 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.107338622, 2024-09-16T04:09:08.709 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.710 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:09:08.711 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.712 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.713 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 46.026416069, 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.714 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:09:08.715 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.716 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.717 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.866012011000002, 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.718 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.719 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.720 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:09:08.721 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 41.189907173999998, 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.722 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:08.723 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.724 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.725 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.726 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.727 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 40.168194149000001, 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.728 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:09:08.729 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:09:08.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.731 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:09:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 40.168141921999997, 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:09:08.733 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.734 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.735 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:09:08.736 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:09:08.737 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.11s6 68/56 ECSubWrite(tid=17, reqid=osd.10.0:17, at_version=68'19, trim_to=0'0, roll_forward_to=64'17))", 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.158456000000001, 2024-09-16T04:09:08.738 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014242092, 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:08.739 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8500000000000001e-07 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539199+0000", 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4397999999999999e-05 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.740 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539248+0000", 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9033000000000003e-05 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539258+0000", 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8800000000000003e-06 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539858+0000", 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060046199999999996 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.741 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539867+0000", 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.363e-06 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553388+0000", 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013520249 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553416+0000", 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8121999999999999e-05 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.742 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=64, reqid=osd.4.0:64, at_version=68'43, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.157341996, 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014089379000000001, 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:08.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1500000000000001e-07 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540308+0000", 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0585999999999999e-05 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540337+0000", 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8560000000000001e-05 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540345+0000", 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6299999999999998e-06 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540400+0000", 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4818000000000003e-05 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540411+0000", 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1009000000000001e-05 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554349+0000", 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013938394 2024-09-16T04:09:08.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554377+0000", 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8167e-05 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=66, reqid=osd.4.0:66, at_version=68'45, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.155500705000001, 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01391278, 2024-09-16T04:09:08.747 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542130+0000", 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6600000000000005e-07 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542154+0000", 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3862999999999998e-05 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542183+0000", 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9108999999999999e-05 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542192+0000", 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549962+0000", 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007769977 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549972+0000", 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0074e-05 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556017+0000", 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060444319999999998 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556042+0000", 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5012999999999999e-05 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=119, reqid=client.4440.0:705, at_version=73'60, trim_to=0'0, roll_forward_to=73'57))", 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.202944630000001, 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035584957, 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494686+0000", 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9999999999999996e-07 2024-09-16T04:09:08.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494705+0000", 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9208999999999999e-05 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529770+0000", 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035064924999999997 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529777+0000", 2024-09-16T04:09:08.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5069999999999997e-06 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529831+0000", 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4122000000000001e-05 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529845+0000", 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4571e-05 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530243+0000", 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039740100000000003 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530270+0000", 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7421999999999999e-05 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.756 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:708 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 69] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.18587509, 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49730772000000001, 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 708 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511756+0000", 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6029999999999999e-06 2024-09-16T04:09:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511762+0000", 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2500000000000003e-06 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511774+0000", 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1761e-05 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511797+0000", 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2416999999999999e-05 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511872+0000", 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5437000000000003e-05 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511975+0000", 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010341899999999999 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.526578+0000", 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014602084 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529422+0000", 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028444880000000001 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008984+0000", 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47956190999999998 2024-09-16T04:09:08.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009062+0000", 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8350999999999996e-05 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:709 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.185819588000001, 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017140263999999999, 2024-09-16T04:09:08.762 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 709 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0800000000000001e-07 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511812+0000", 2024-09-16T04:09:08.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7799999999999999e-06 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511816+0000", 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.172e-06 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511819+0000", 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9739999999999998e-06 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512187+0000", 2024-09-16T04:09:08.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036796799999999998 2024-09-16T04:09:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512240+0000", 2024-09-16T04:09:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3013999999999998e-05 2024-09-16T04:09:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528792+0000", 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016551461999999999 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528863+0000", 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0797000000000007e-05 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528950+0000", 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7788999999999996e-05 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.767 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.167003072, 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.36295003300000001, 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:08.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7899999999999998e-07 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530637+0000", 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892881+0000", 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.362243749 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892890+0000", 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5620000000000004e-06 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892950+0000", 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0270999999999999e-05 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892959+0000", 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1689999999999998e-06 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893539+0000", 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000579628 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893577+0000", 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8056999999999998e-05 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=131, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 20.166718535000001, 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47861527599999998, 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.772 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530912+0000", 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4400000000000001e-07 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530915+0000", 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5530000000000001e-06 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008849+0000", 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47793379000000002 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008854+0000", 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0340000000000004e-06 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008927+0000", 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3088000000000002e-05 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009526+0000", 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059936699999999996 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=132, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 19.687299265, 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44531548500000001, 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010331+0000", 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9600000000000001e-07 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010332+0000", 2024-09-16T04:09:08.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.454e-06 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455554+0000", 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44522236199999998 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455561+0000", 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4189999999999999e-06 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455608+0000", 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6928000000000002e-05 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455646+0000", 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8126000000000001e-05 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:819 3.9s0 3:94de6fe1:::smithi01639990-3:head [stat,delete] snapc 1e=[1e,1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:58.510155+0000", 2024-09-16T04:09:08.779 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.187474594999999, 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016397412, 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 819 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510155+0000", 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510155+0000", 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510157+0000", 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.823e-06 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510167+0000", 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0237e-05 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510184+0000", 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7007000000000002e-05 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510193+0000", 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5050000000000007e-06 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510261+0000", 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8507e-05 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510270+0000", 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8890000000000008e-06 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.516074+0000", 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058035379999999996 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.516179+0000", 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010501 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.518371+0000", 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021920770000000002 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.524696+0000", 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063251929999999998 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.526496+0000", 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001800486 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.526553+0000", 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6140000000000001e-05 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 92/56 ECSubWriteReply(tid=194, last_complete=92'115, committed=1, applied=1))", 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.490709+0000", 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.2069202830000001, 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59931532099999996, 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.490709+0000", 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.490709+0000", 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.490710+0000", 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4099999999999998e-07 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.490714+0000", 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5319999999999998e-06 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.089874+0000", 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59916044499999999 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.089888+0000", 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3696999999999999e-05 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.089976+0000", 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8580999999999998e-05 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.090025+0000", 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8325000000000002e-05 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 92/56 ECSubWriteReply(tid=194, last_complete=92'115, committed=1, applied=1))", 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.494594+0000", 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.2030358240000001, 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.38089667199999999, 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:08.789 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.494594+0000", 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.494594+0000", 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.494595+0000", 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.494600+0000", 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9899999999999997e-06 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.875394+0000", 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.38079475000000002 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.875401+0000", 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2820000000000002e-06 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.875474+0000", 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3679999999999999e-05 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.875491+0000", 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6093999999999999e-05 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.792 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:08.793 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:08.793 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:08.794 INFO:tasks.rados.rados.0.smithi016.stdout:387: finishing copy_from racing read to smithi01639990-31 2024-09-16T04:09:08.794 INFO:tasks.rados.rados.0.smithi016.stdout:387: finishing copy_from to smithi01639990-31 2024-09-16T04:09:08.794 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 31 v 37 (ObjNum 137 snap 28 seq_num 137) dirty exists 2024-09-16T04:09:08.794 INFO:tasks.rados.rados.0.smithi016.stdout:382: done (6 left) 2024-09-16T04:09:08.795 INFO:tasks.rados.rados.0.smithi016.stdout:386: done (5 left) 2024-09-16T04:09:08.795 INFO:tasks.rados.rados.0.smithi016.stdout:387: done (4 left) 2024-09-16T04:09:08.795 INFO:tasks.rados.rados.0.smithi016.stdout:388: done (3 left) 2024-09-16T04:09:08.795 INFO:tasks.rados.rados.0.smithi016.stdout:389: done (2 left) 2024-09-16T04:09:08.795 INFO:tasks.rados.rados.0.smithi016.stdout:390: done (1 left) 2024-09-16T04:09:08.795 INFO:tasks.rados.rados.0.smithi016.stdout:391: done (0 left) 2024-09-16T04:09:08.795 INFO:tasks.rados.rados.0.smithi016.stdout:392: snap_create 2024-09-16T04:09:08.795 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:09:08.792+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:09:08.796 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:09:08.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T04:09:08.896 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:09:08.893+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:09:08.939 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:08.939 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:08.939 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:09:08.939 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:08.948 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:09:08.996 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:09:08.995+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:09:09.050 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:09.050 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:09.050 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:09:09.050 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:09:09.050 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:09.058 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:09:09.098 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:09.096+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-16T04:09:09.119+0000 7f5f0b59e640 0 --2- 172.21.15.16:0/594037918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f0c149e90 0x7f5f0c1dedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:09:09.146 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:09:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:09:09.164 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:09.164 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:09:09.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:09:09.164 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:09:09.164 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.164 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:09:09.164 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:09:09.164 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 62.067266355000001, 2024-09-16T04:09:09.164 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:09:09.165 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.166 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:09:09.167 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.168 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 50.731749385000001, 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.169 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:09:09.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:09:09.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:09:09.172 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 50.271799158, 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:09:09.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:09:09.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:09:09.177 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.948286553000003, 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.bs7 57/56 [PushOp(3:d683cb60:::smithi01639990-49:head, version: 53'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d683cb60:::smithi01639990-49:head@53'38, size: 2310144, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:09:09.181 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 36.600806056000003, 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.043702234999999999, 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:09:09.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560769+0000", 2024-09-16T04:09:09.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0049999999999999e-06 2024-09-16T04:09:09.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.561301+0000", 2024-09-16T04:09:09.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053276500000000004 2024-09-16T04:09:09.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.185 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.185 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603955+0000", 2024-09-16T04:09:09.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042653605999999997 2024-09-16T04:09:09.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603967+0000", 2024-09-16T04:09:09.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2065000000000001e-05 2024-09-16T04:09:09.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604063+0000", 2024-09-16T04:09:09.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5599000000000004e-05 2024-09-16T04:09:09.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604470+0000", 2024-09-16T04:09:09.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040719500000000001 2024-09-16T04:09:09.187 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.187 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.187 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=252, reqid=osd.3.0:252, at_version=57'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 36.252079916, 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27497091299999998, 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:09.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:09.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909517+0000", 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2886000000000001e-05 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.190 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183847+0000", 2024-09-16T04:09:09.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27432981400000001 2024-09-16T04:09:09.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.191 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.191 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.191 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183853+0000", 2024-09-16T04:09:09.191 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2929999999999997e-06 2024-09-16T04:09:09.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183902+0000", 2024-09-16T04:09:09.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9008999999999998e-05 2024-09-16T04:09:09.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183911+0000", 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184435+0000", 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052493699999999995 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184465+0000", 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9292e-05 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.193 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=253, reqid=osd.3.0:253, at_version=57'67, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 36.251451885000002, 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50851210899999999, 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910123+0000", 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910136+0000", 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.331e-05 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416910+0000", 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50677369000000005 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416919+0000", 2024-09-16T04:09:09.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8290000000000003e-06 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416975+0000", 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5983999999999998e-05 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416985+0000", 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418599+0000", 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001614515 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418634+0000", 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4588999999999997e-05 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.198 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 64/56 ECSubWrite(tid=285, reqid=client.4440.0:535, at_version=0'0, trim_to=0'0, roll_forward_to=64'72))", 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.620004816000002, 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012095003, 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:09:09.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541570+0000", 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.173e-06 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541582+0000", 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2106999999999999e-05 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552877+0000", 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011294692 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552889+0000", 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1875000000000001e-05 2024-09-16T04:09:09.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552958+0000", 2024-09-16T04:09:09.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9510000000000004e-05 2024-09-16T04:09:09.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552967+0000", 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6240000000000008e-06 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553631+0000", 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066445299999999996 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553664+0000", 2024-09-16T04:09:09.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2568999999999999e-05 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 68/46 ECSubWrite(tid=65, reqid=osd.4.0:65, at_version=68'22, trim_to=0'0, roll_forward_to=67'20))", 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.620278557999999, 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010787494999999999, 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.205 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:09.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541313+0000", 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7431e-05 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541369+0000", 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6311999999999999e-05 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541376+0000", 2024-09-16T04:09:09.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5729999999999996e-06 2024-09-16T04:09:09.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544486+0000", 2024-09-16T04:09:09.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031104150000000001 2024-09-16T04:09:09.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544495+0000", 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0960000000000007e-06 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552010+0000", 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075148200000000002 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552083+0000", 2024-09-16T04:09:09.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2661000000000001e-05 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=338, reqid=osd.3.0:338, at_version=69'72, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.617186610000001, 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010064081000000001, 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.210 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544388+0000", 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9800000000000003e-07 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544409+0000", 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0993e-05 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544442+0000", 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2937e-05 2024-09-16T04:09:09.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544449+0000", 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.588e-06 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547347+0000", 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028976599999999998 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547357+0000", 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6479999999999998e-06 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554419+0000", 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070620819999999999 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554451+0000", 2024-09-16T04:09:09.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2475000000000003e-05 2024-09-16T04:09:09.215 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.215 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.215 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.216 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s6 69/56 ECSubWrite(tid=202, reqid=osd.7.0:202, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:09.216 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:09:09.216 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.615849815000001, 2024-09-16T04:09:09.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010752746000000001, 2024-09-16T04:09:09.216 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.216 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.216 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545725+0000", 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4199999999999995e-07 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545750+0000", 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4915000000000001e-05 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546880+0000", 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011307870000000001 2024-09-16T04:09:09.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546891+0000", 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0825000000000001e-05 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551495+0000", 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004604131 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551504+0000", 2024-09-16T04:09:09.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6230000000000003e-06 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556450+0000", 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049460210000000001 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556477+0000", 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6801999999999999e-05 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 82/56 ECSubWrite(tid=197, reqid=client.4440.0:785, at_version=82'40, trim_to=0'0, roll_forward_to=74'38))", 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 11.668807178, 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0095796900000000001, 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6400000000000001e-07 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492863+0000", 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6371000000000003e-05 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492883+0000", 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9507000000000001e-05 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492894+0000", 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0998999999999999e-05 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492958+0000", 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3898000000000004e-05 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492968+0000", 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0012e-05 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.502307+0000", 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093395509999999998 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.502346+0000", 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9187999999999998e-05 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.17s2 83/56 ECSubWrite(tid=206, reqid=client.4440.0:818, at_version=0'0, trim_to=0'0, roll_forward_to=83'73))", 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:58.510113+0000", 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 10.651460725, 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.57355782399999999, 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.225 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510113+0000", 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510113+0000", 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510114+0000", 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.68e-07 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510124+0000", 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0117e-05 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082885+0000", 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.57276119999999997 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.227 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082895+0000", 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7580000000000004e-06 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082959+0000", 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4545e-05 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082968+0000", 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9250000000000001e-06 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.083638+0000", 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00067024900000000004 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.083671+0000", 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2462000000000002e-05 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s5 92/56 ECSubWrite(tid=226, reqid=osd.0.0:226, at_version=92'36, trim_to=0'0, roll_forward_to=89'34))", 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.452733+0000", 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.708841244, 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010158257, 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452733+0000", 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452733+0000", 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452733+0000", 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1299999999999999e-07 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452766+0000", 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3034000000000003e-05 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452853+0000", 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6799999999999996e-05 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452867+0000", 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4837e-05 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452932+0000", 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4726000000000003e-05 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452942+0000", 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0132999999999999e-05 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.462867+0000", 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.009924782 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.462891+0000", 2024-09-16T04:09:09.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3731999999999999e-05 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 92/56 ECSubWrite(tid=440, reqid=osd.3.0:440, at_version=92'95, trim_to=0'0, roll_forward_to=89'93))", 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.453317+0000", 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.7082570459999999, 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011870421000000001, 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.234 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453317+0000", 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453317+0000", 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453318+0000", 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1149999999999999e-06 2024-09-16T04:09:09.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453343+0000", 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5352000000000001e-05 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453514+0000", 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017035999999999999 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453523+0000", 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2350000000000005e-06 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453600+0000", 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7463e-05 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453611+0000", 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0648e-05 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.465151+0000", 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011540478999999999 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.465187+0000", 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5769000000000003e-05 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.238 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.4s2 92/56 ECSubWrite(tid=53, reqid=osd.10.0:53, at_version=92'41, trim_to=0'0, roll_forward_to=89'39))", 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.453423+0000", 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.7081503009999999, 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0095119709999999993, 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453423+0000", 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453423+0000", 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453424+0000", 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0900000000000001e-07 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453452+0000", 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8013000000000001e-05 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.454574+0000", 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00112243 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.454583+0000", 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8240000000000006e-06 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.454655+0000", 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1883000000000003e-05 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.454664+0000", 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9770000000000006e-06 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.462901+0000", 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082373819999999997 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.462935+0000", 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4252999999999999e-05 2024-09-16T04:09:09.242 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.15s7 92/48 ECSubWrite(tid=190, reqid=osd.8.0:190, at_version=92'83, trim_to=0'0, roll_forward_to=89'81))", 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.459701+0000", 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.7018728990000001, 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.026074818, 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.243 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.459701+0000", 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.459701+0000", 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.459701+0000", 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:09:09.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.459720+0000", 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8476999999999999e-05 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.478473+0000", 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.018753216999999999 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.478483+0000", 2024-09-16T04:09:09.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6169999999999996e-06 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.478577+0000", 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4059000000000002e-05 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.478589+0000", 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2663999999999999e-05 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.485750+0000", 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007160596 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.485776+0000", 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5598000000000001e-05 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 92/56 ECSubWrite(tid=192, reqid=osd.8.0:192, at_version=92'115, trim_to=0'0, roll_forward_to=92'113))", 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.479459+0000", 2024-09-16T04:09:09.247 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 1.6821147939999999, 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0095768099999999998, 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.479459+0000", 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.479459+0000", 2024-09-16T04:09:09.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.479460+0000", 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9100000000000003e-07 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.479476+0000", 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6595999999999998e-05 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.486477+0000", 2024-09-16T04:09:09.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070005110000000001 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.486488+0000", 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1017e-05 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.486549+0000", 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1069999999999996e-05 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.486560+0000", 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0674e-05 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.488988+0000", 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0024284210000000001 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.489036+0000", 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7830000000000001e-05 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as5 93/56 ECSubWrite(tid=81, reqid=osd.6.0:81, at_version=93'67, trim_to=0'0, roll_forward_to=92'65))", 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:08.457204+0000", 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.70436994600000002, 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.036487762, 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.457204+0000", 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.457204+0000", 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.457204+0000", 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2699999999999999e-07 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.457228+0000", 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3234000000000002e-05 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.487257+0000", 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.03002954 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.487266+0000", 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9430000000000006e-06 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.487335+0000", 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9196000000000006e-05 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.487350+0000", 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4561e-05 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.493666+0000", 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063165119999999998 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.493692+0000", 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5148999999999999e-05 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as5 93/56 ECSubWrite(tid=83, reqid=client.4440.0:930, at_version=0'0, trim_to=0'0, roll_forward_to=93'69))", 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:08.496719+0000", 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 0.66485477299999995, 2024-09-16T04:09:09.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.060932394000000001, 2024-09-16T04:09:09.257 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:09.257 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:09.257 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:09.257 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:09.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.496719+0000", 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.496719+0000", 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.496720+0000", 2024-09-16T04:09:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.92e-07 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.496730+0000", 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0077e-05 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.556875+0000", 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.060145123000000002 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.556885+0000", 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8980000000000008e-06 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.556947+0000", 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2687000000000005e-05 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.556957+0000", 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9899999999999992e-06 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.557615+0000", 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00065753600000000001 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.557651+0000", 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6390999999999998e-05 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:09.261 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:09.262 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:09:09.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:09:09.263 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:09:09.264 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:09:09.264 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:09:09.265 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:09:09.265 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:09:09.265 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:09:09.265 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:09:09.266 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:09:09.266 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:09:09.266 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:09.261+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:09.273 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:09:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:09:09.294 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:09:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:09:09.315 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:09:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:09:09.337 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:09:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:09:09.358 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:09:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:09:09.364 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:09:09.363+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:09:09.370 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:09.370 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:09.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:09:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:09:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:09:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-16T04:09:09.444 INFO:tasks.rados.rados.0.smithi016.stdout:392: done (0 left) 2024-09-16T04:09:09.444 INFO:tasks.rados.rados.0.smithi016.stdout:393: snap_create 2024-09-16T04:09:09.465 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:09:09.464+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:09:09.483 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:09.483 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:09.491 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:09:09.566 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:09.565+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:09.594 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:09.595 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:09.603 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:09:09.667 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:09.665+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:09.707 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:09.707 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:09.715 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:09:09.767 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:09:09.766+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:09:09.858 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:09.858 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:09.867 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:09:09.869 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:09:09.867+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:09:09.969 INFO:teuthology.orchestra.run.smithi194.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:09.969 INFO:teuthology.orchestra.run.smithi194.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:09.970 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:09.968+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:09.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:09:10.070 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:10.069+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:10.091 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:10.092 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:10.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:09:10.172 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:09:10.170+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:09:10.207 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:10.208 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:10.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:09:10.273 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:09:10.271+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:09:10.323 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:10.323 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:10.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T04:09:10.373 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:10.372+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:10.439 INFO:tasks.rados.rados.0.smithi016.stdout:393: done (0 left) 2024-09-16T04:09:10.439 INFO:tasks.rados.rados.0.smithi016.stdout:append oid 16 current snap is 35 2024-09-16T04:09:10.439 INFO:tasks.rados.rados.0.smithi016.stdout:394: seq_num 160 ranges {2080768=294912} 2024-09-16T04:09:10.452 INFO:tasks.rados.rados.0.smithi016.stdout:394: writing smithi01639990-16 from 2080768 to 2375680 tid 1 2024-09-16T04:09:10.452 INFO:tasks.rados.rados.0.smithi016.stdout:395: setattr oid 37 current snap is 35 2024-09-16T04:09:10.454 INFO:tasks.rados.rados.0.smithi016.stdout:396: read oid 36 snap 32 2024-09-16T04:09:10.454 INFO:tasks.rados.rados.0.smithi016.stdout:396: expect deleted 2024-09-16T04:09:10.455 INFO:tasks.rados.rados.0.smithi016.stdout:397: read oid 23 snap 26 2024-09-16T04:09:10.455 INFO:tasks.rados.rados.0.smithi016.stdout:397: expect (ObjNum 104 snap 17 seq_num 104) 2024-09-16T04:09:10.455 INFO:tasks.rados.rados.0.smithi016.stdout:398: read oid 34 snap -1 2024-09-16T04:09:10.455 INFO:tasks.rados.rados.0.smithi016.stdout:398: expect deleted 2024-09-16T04:09:10.455 INFO:tasks.rados.rados.0.smithi016.stdout:399: rollback oid 9 current snap is 35 2024-09-16T04:09:10.455 INFO:tasks.rados.rados.0.smithi016.stdout:rollback oid 9 to 33 2024-09-16T04:09:10.455 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 6 2024-09-16T04:09:10.455 INFO:tasks.rados.rados.0.smithi016.stdout:396: done (5 left) 2024-09-16T04:09:10.455 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 5 2024-09-16T04:09:10.464 INFO:tasks.rados.rados.0.smithi016.stdout:399: finishing rollback tid 0 to smithi01639990-9 2024-09-16T04:09:10.465 INFO:tasks.rados.rados.0.smithi016.stdout:399: finishing rollback tid 1 to smithi01639990-9 2024-09-16T04:09:10.465 INFO:tasks.rados.rados.0.smithi016.stdout:398: done (4 left) 2024-09-16T04:09:10.465 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 4 2024-09-16T04:09:10.465 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 37 v 121 (ObjNum 49 snap 0 seq_num 1768186217) dirty exists 2024-09-16T04:09:10.465 INFO:tasks.rados.rados.0.smithi016.stdout:395: done (3 left) 2024-09-16T04:09:10.466 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 3 2024-09-16T04:09:10.466 INFO:tasks.rados.rados.0.smithi016.stdout:394: finishing write tid 1 to smithi01639990-16 2024-09-16T04:09:10.467 INFO:tasks.rados.rados.0.smithi016.stdout:399: finishing rollback tid 2 to smithi01639990-9 2024-09-16T04:09:10.467 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 9 v 106 (ObjNum 118 snap 20 seq_num 118) dirty exists 2024-09-16T04:09:10.467 INFO:tasks.rados.rados.0.smithi016.stdout:399: done (2 left) 2024-09-16T04:09:10.467 INFO:tasks.rados.rados.0.smithi016.stdout: waiting on 2 2024-09-16T04:09:10.477 INFO:tasks.rados.rados.0.smithi016.stdout:394: finishing write tid 2 to smithi01639990-16 2024-09-16T04:09:10.477 INFO:tasks.rados.rados.0.smithi016.stdout:397: expect (ObjNum 104 snap 17 seq_num 104) 2024-09-16T04:09:10.478 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:10.476+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:10.487 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:10.487 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:10.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T04:09:10.521 INFO:tasks.rados.rados.0.smithi016.stdout:394: finishing write tid 3 to smithi01639990-16 2024-09-16T04:09:10.521 INFO:tasks.rados.rados.0.smithi016.stdout:394: oid 16 updating version 0 to 121 2024-09-16T04:09:10.521 INFO:tasks.rados.rados.0.smithi016.stdout:394: oid 16 updating version 121 to 122 2024-09-16T04:09:10.521 INFO:tasks.rados.rados.0.smithi016.stdout:update_object_version oid 16 v 122 (ObjNum 160 snap 35 seq_num 160) dirty exists 2024-09-16T04:09:10.522 INFO:tasks.rados.rados.0.smithi016.stdout:394: left oid 16 (ObjNum 160 snap 35 seq_num 160) 2024-09-16T04:09:10.522 INFO:tasks.rados.rados.0.smithi016.stdout:394: done (1 left) 2024-09-16T04:09:10.522 INFO:tasks.rados.rados.0.smithi016.stdout:397: done (0 left) 2024-09-16T04:09:10.526 INFO:tasks.rados.rados.0.smithi016.stderr:0 errors. 2024-09-16T04:09:10.526 INFO:tasks.rados.rados.0.smithi016.stderr: 2024-09-16T04:09:10.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:09:10.578 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1] 2024-09-16T04:09:10.578 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T04:09:10.578 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T04:09:10.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T04:09:10.605 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:10.605 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:10.606 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-16T04:09:10.603+0000 7ffa94926640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33063) UID: 0 2024-09-16T04:09:10.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T04:09:10.681 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:10.679+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:10.766 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:10.766 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:10.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T04:09:10.782 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-09-16T04:09:10.781+0000 7f6528b4c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33137) UID: 0 2024-09-16T04:09:10.885 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:10.885 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:10.885 INFO:tasks.ceph.osd.11.smithi194.stderr:2024-09-16T04:09:10.884+0000 7f3193aac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33297) UID: 0 2024-09-16T04:09:10.888 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:10.888 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"fsid":"44381953-4509-4b15-853e-3887928dfa29","created":"2024-09-16T04:07:36.650572+0000","modified":"2024-09-16T04:09:10.434988+0000","last_up_change":"2024-09-16T04:08:51.432862+0000","last_in_change":"2024-09-16T04:08:31.429435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:07:48.723300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:07:49.309634+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-16T04:08:01.082632+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":8,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"46","auid":0,"snap_mode":"selfmanaged","snap_seq":36,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lrcprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"4506422f-7ccf-46d7-af66-62e8cace167a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6825","nonce":3110165165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6827","nonce":3110165165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6831","nonce":3110165165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6829","nonce":3110165165}]},"public_addr":"172.21.15.16:6825/3110165165","cluster_addr":"172.21.15.16:6827/3110165165","heartbeat_back_addr":"172.21.15.16:6831/3110165165","heartbeat_front_addr":"172.21.15.16:6829/3110165165","state":["exists","up"]},{"osd":1,"uuid":"8dc0488e-efb6-4664-a799-5e14c56769b0","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":54,"up_from":76,"up_thru":48,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6810","nonce":4190982331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6814","nonce":4190982331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6822","nonce":4190982331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6818","nonce":4190982331}]},"public_addr":"172.21.15.16:6810/4190982331","cluster_addr":"172.21.15.16:6814/4190982331","heartbeat_back_addr":"172.21.15.16:6822/4190982331","heartbeat_front_addr":"172.21.15.16:6818/4190982331","state":["exists","up"]},{"osd":2,"uuid":"1686e657-c1f3-499b-8b10-67f428c5c216","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6801","nonce":2266032822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6803","nonce":2266032822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6807","nonce":2266032822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6805","nonce":2266032822}]},"public_addr":"172.21.15.16:6801/2266032822","cluster_addr":"172.21.15.16:6803/2266032822","heartbeat_back_addr":"172.21.15.16:6807/2266032822","heartbeat_front_addr":"172.21.15.16:6805/2266032822","state":["exists","up"]},{"osd":3,"uuid":"730ad37c-1f5f-4d23-a1a7-c75320506665","up":1,"in":1,"weight":0.95001220703125,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6811","nonce":3389562884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6815","nonce":3389562884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6823","nonce":3389562884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6819","nonce":3389562884}]},"public_addr":"172.21.15.16:6811/3389562884","cluster_addr":"172.21.15.16:6815/3389562884","heartbeat_back_addr":"172.21.15.16:6823/3389562884","heartbeat_front_addr":"172.21.15.16:6819/3389562884","state":["exists","up"]},{"osd":4,"uuid":"d82c5c44-72fe-4680-9c04-ab4e0775b929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6811","nonce":1733996396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6813","nonce":1733996396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6817","nonce":1733996396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6815","nonce":1733996396}]},"public_addr":"172.21.15.174:6811/1733996396","cluster_addr":"172.21.15.174:6813/1733996396","heartbeat_back_addr":"172.21.15.174:6817/1733996396","heartbeat_front_addr":"172.21.15.174:6815/1733996396","state":["exists","up"]},{"osd":5,"uuid":"41502441-b0d9-460f-92c0-b8b3a915cac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6827","nonce":2956753716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6829","nonce":2956753716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6833","nonce":2956753716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6831","nonce":2956753716}]},"public_addr":"172.21.15.174:6827/2956753716","cluster_addr":"172.21.15.174:6829/2956753716","heartbeat_back_addr":"172.21.15.174:6833/2956753716","heartbeat_front_addr":"172.21.15.174:6831/2956753716","state":["exists","up"]},{"osd":6,"uuid":"45cc7521-11db-4496-8d0c-1abc42962810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6803","nonce":2727832023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6805","nonce":2727832023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6809","nonce":2727832023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6807","nonce":2727832023}]},"public_addr":"172.21.15.174:6803/2727832023","cluster_addr":"172.21.15.174:6805/2727832023","heartbeat_back_addr":"172.21.15.174:6809/2727832023","heartbeat_front_addr":"172.21.15.174:6807/2727832023","state":["exists","up"]},{"osd":7,"uuid":"1b5d6337-73c3-494b-91cd-fcfa7d307050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6819","nonce":2338392542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6821","nonce":2338392542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6825","nonce":2338392542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6823","nonce":2338392542}]},"public_addr":"172.21.15.174:6819/2338392542","cluster_addr":"172.21.15.174:6821/2338392542","heartbeat_back_addr":"172.21.15.174:6825/2338392542","heartbeat_front_addr":"172.21.15.174:6823/2338392542","state":["exists","up"]},{"osd":8,"uuid":"766efffa-3aff-4aa3-9cf8-88c5c7986d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621487148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621487148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621487148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621487148}]},"public_addr":"172.21.15.194:6801/2621487148","cluster_addr":"172.21.15.194:6803/2621487148","heartbeat_back_addr":"172.21.15.194:6807/2621487148","heartbeat_front_addr":"172.21.15.194:6805/2621487148","state":["exists","up"]},{"osd":9,"uuid":"755cc9aa-2f6f-4d7a-84cf-7feb74d65c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6817","nonce":367304498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6819","nonce":367304498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6823","nonce":367304498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6821","nonce":367304498}]},"public_addr":"172.21.15.194:6817/367304498","cluster_addr":"172.21.15.194:6819/367304498","heartbeat_back_addr":"172.21.15.194:6823/367304498","heartbeat_front_addr":"172.21.15.194:6821/367304498","state":["exists","up"]},{"osd":10,"uuid":"76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6825","nonce":3349503344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6827","nonce":3349503344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6831","nonce":3349503344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6829","nonce":3349503344}]},"public_addr":"172.21.15.194:6825/3349503344","cluster_addr":"172.21.15.194:6827/3349503344","heartbeat_back_addr":"172.21.15.194:6831/3349503344","heartbeat_front_addr":"172.21.15.194:6829/3349503344","state":["exists","up"]},{"osd":11,"uuid":"f71641eb-1331-4b39-9c0e-bf98e91ed295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6809","nonce":2531896090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6811","nonce":2531896090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6815","nonce":2531896090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6813","nonce":2531896090}]},"public_addr":"172.21.15.194:6809/2531896090","cluster_addr":"172.21.15.194:6811/2531896090","heartbeat_back_addr":"172.21.15.194:6815/2531896090","heartbeat_front_addr":"172.21.15.194:6813/2531896090","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:01.680673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T04:08:58.907614+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:47.992191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:48.214554+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:07:46.162300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:59.631647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:58.489685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:08.502081+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:48.771857+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:06.156463+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:56.419990+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:00.745572+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.d","osds":[7,8,4,11,9,0,1,5]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":22,"length":1},{"begin":27,"length":1},{"begin":31,"length":1},{"begin":33,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":30,"length":1},{"begin":32,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:09:10.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T04:09:10.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:09:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:10.920 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 95, 2024-09-16T04:09:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "44381953-4509-4b15-853e-3887928dfa29", 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2024-09-16T04:07:36.650572+0000", 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2024-09-16T04:09:10.434988+0000", 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2024-09-16T04:08:51.432862+0000", 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2024-09-16T04:08:31.429435+0000", 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5799936, 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:09:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 8, 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 3, 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 12, 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "squid", 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_crimson": false, 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": ".mgr", 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-16T04:07:48.723300+0000", 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "is_stretch_pool": false, 2024-09-16T04:09:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-16T04:09:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "21", 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-16T04:09:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-16T04:09:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-16T04:09:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": {} 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "score_type": "Fair distribution", 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 11.760000228881836, 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 11.760000228881836, 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.17000000178813934, 2024-09-16T04:09:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2, 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2, 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-16T04:07:49.309634+0000", 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2024-09-16T04:09:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "is_stretch_pool": false, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2024-09-16T04:09:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "23", 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T04:09:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 2, 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 23, 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-16T04:09:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-16T04:09:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "read_balance": { 2024-09-16T04:09:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "score_type": "Fair distribution", 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "score_acting": 2.9900000095367432, 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "score_stable": 2.9900000095367432, 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "optimal_score": 0.67000001668930054, 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_acting": 2, 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "raw_score_stable": 2, 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity": 1, 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "average_primary_affinity_weighted": 1 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 3, 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "unique_pool_0", 2024-09-16T04:09:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2024-09-16T04:08:01.082632+0000", 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 3, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 8, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 5, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 1, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "is_stretch_pool": false, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2024-09-16T04:09:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 26, 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 26, 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 26, 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 26, 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 26, 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2024-09-16T04:09:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "95", 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "46", 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 36, 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 95, 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2024-09-16T04:09:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "lrcprofile", 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2024-09-16T04:09:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 16384, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2024-09-16T04:09:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "rados": {} 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4506422f-7ccf-46d7-af66-62e8cace167a", 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 17, 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 56, 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6824", 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6825", 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6826", 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:09:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6827", 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6830", 2024-09-16T04:09:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6831", 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6828", 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6829", 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3110165165 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6825/3110165165", 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6827/3110165165", 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6831/3110165165", 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6829/3110165165", 2024-09-16T04:09:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "8dc0488e-efb6-4664-a799-5e14c56769b0", 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 0, 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0, 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 18, 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 54, 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 76, 2024-09-16T04:09:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 48, 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 55, 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4190982331 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2024-09-16T04:09:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4190982331 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4190982331 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4190982331 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4190982331 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2024-09-16T04:09:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4190982331 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4190982331 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2024-09-16T04:09:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4190982331 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6810/4190982331", 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6814/4190982331", 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6822/4190982331", 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6818/4190982331", 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "1686e657-c1f3-499b-8b10-67f428c5c216", 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 20, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2266032822 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/2266032822", 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/2266032822", 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/2266032822", 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/2266032822", 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "730ad37c-1f5f-4d23-a1a7-c75320506665", 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0.95001220703125, 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 56, 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3389562884 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6811/3389562884", 2024-09-16T04:09:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6815/3389562884", 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/3389562884", 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6819/3389562884", 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d82c5c44-72fe-4680-9c04-ab4e0775b929", 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 48, 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6810", 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6811", 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6812", 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6813", 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6816", 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6817", 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6814", 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6815", 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1733996396 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.174:6811/1733996396", 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.174:6813/1733996396", 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.174:6817/1733996396", 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.174:6815/1733996396", 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "41502441-b0d9-460f-92c0-b8b3a915cac5", 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 56, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6826", 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6827", 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6828", 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6829", 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6832", 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6833", 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6830", 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6831", 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2956753716 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.174:6827/2956753716", 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.174:6829/2956753716", 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.174:6833/2956753716", 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.174:6831/2956753716", 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 6, 2024-09-16T04:09:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "45cc7521-11db-4496-8d0c-1abc42962810", 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 19, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 56, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6802", 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6803", 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6804", 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6805", 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6808", 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6809", 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6806", 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6807", 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2727832023 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.174:6803/2727832023", 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.174:6805/2727832023", 2024-09-16T04:09:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.174:6809/2727832023", 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.174:6807/2727832023", 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 7, 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "1b5d6337-73c3-494b-91cd-fcfa7d307050", 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 56, 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6818", 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6819", 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6820", 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:09:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6821", 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6824", 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:09:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6825", 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6822", 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:09:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.174:6823", 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2338392542 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.174:6819/2338392542", 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.174:6821/2338392542", 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.174:6825/2338392542", 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.174:6823/2338392542", 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 8, 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "766efffa-3aff-4aa3-9cf8-88c5c7986d21", 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 56, 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6800", 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6801", 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6802", 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6803", 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6806", 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6807", 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6804", 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6805", 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2621487148 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.194:6801/2621487148", 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.194:6803/2621487148", 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.194:6807/2621487148", 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.194:6805/2621487148", 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 9, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "755cc9aa-2f6f-4d7a-84cf-7feb74d65c48", 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6816", 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6817", 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6818", 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6819", 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6822", 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6823", 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6820", 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6821", 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 367304498 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.194:6817/367304498", 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.194:6819/367304498", 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.194:6823/367304498", 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.194:6821/367304498", 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 10, 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377", 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 56, 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6824", 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:09:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6825", 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6826", 2024-09-16T04:09:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6827", 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6830", 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6831", 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6828", 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6829", 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3349503344 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.194:6825/3349503344", 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.194:6827/3349503344", 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.194:6831/3349503344", 2024-09-16T04:09:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.194:6829/3349503344", 2024-09-16T04:09:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 11, 2024-09-16T04:09:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "f71641eb-1331-4b39-9c0e-bf98e91ed295", 2024-09-16T04:09:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 18, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 56, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6808", 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6809", 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6810", 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6811", 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6814", 2024-09-16T04:09:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6815", 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6812", 2024-09-16T04:09:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.194:6813", 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2531896090 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.194:6809/2531896090", 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.194:6811/2531896090", 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.194:6815/2531896090", 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.194:6813/2531896090", 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2024-09-16T04:09:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:09:01.680673+0000", 2024-09-16T04:09:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 65536, 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:08:58.907614+0000", 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 55 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2024-09-16T04:09:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:08:47.992191+0000", 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:08:48.214554+0000", 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:07:46.162300+0000", 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:08:59.631647+0000", 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 6, 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:08:58.489685+0000", 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 7, 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:09:08.502081+0000", 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 8, 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:08:48.771857+0000", 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 9, 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:09:06.156463+0000", 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 10, 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:08:56.419990+0000", 2024-09-16T04:09:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 11, 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540701547738038271, 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2024-09-16T04:09:00.745572+0000", 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "3.d", 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: 7, 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: 8, 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: 4, 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: 11, 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: 9, 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: 0, 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_primaries": [], 2024-09-16T04:09:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "lrcprofile": { 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-device-class": "", 2024-09-16T04:09:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-num-failure-domains": "0", 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-osds-per-failure-domain": "0", 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-root": "default", 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "4", 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "l": "3", 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "2", 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "lrcprofile", 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "lrc" 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [ 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 3, 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "snaps": [ 2024-09-16T04:09:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "begin": 22, 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "length": 1 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "begin": 27, 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "length": 1 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "begin": 31, 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "length": 1 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "begin": 33, 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "length": 1 2024-09-16T04:09:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [ 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 3, 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "snaps": [ 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "begin": 30, 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "length": 1 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "begin": 32, 2024-09-16T04:09:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "length": 1 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.024 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:11.027 INFO:tasks.thrashosds.thrasher:j is {'epoch': 95, 'fsid': '44381953-4509-4b15-853e-3887928dfa29', 'created': '2024-09-16T04:07:36.650572+0000', 'modified': '2024-09-16T04:09:10.434988+0000', 'last_up_change': '2024-09-16T04:08:51.432862+0000', 'last_in_change': '2024-09-16T04:08:31.429435+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 8, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T04:07:48.723300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T04:07:49.309634+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 23, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T04:08:01.082632+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 8, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '46', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 36, 'snap_epoch': 95, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'lrcprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '4506422f-7ccf-46d7-af66-62e8cace167a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 17, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6824', 'nonce': 3110165165}, {'type': 'v1', 'addr': '172.21.15.16:6825', 'nonce': 3110165165}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6826', 'nonce': 3110165165}, {'type': 'v1', 'addr': '172.21.15.16:6827', 'nonce': 3110165165}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6830', 'nonce': 3110165165}, {'type': 'v1', 'addr': '172.21.15.16:6831', 'nonce': 3110165165}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6828', 'nonce': 3110165165}, {'type': 'v1', 'addr': '172.21.15.16:6829', 'nonce': 3110165165}]}, 'public_addr': '172.21.15.16:6825/3110165165', 'cluster_addr': '172.21.15.16:6827/3110165165', 'heartbeat_back_addr': '172.21.15.16:6831/3110165165', 'heartbeat_front_addr': '172.21.15.16:6829/3110165165', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8dc0488e-efb6-4664-a799-5e14c56769b0', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 54, 'up_from': 76, 'up_thru': 48, 'down_at': 55, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 4190982331}, {'type': 'v1', 'addr': '172.21.15.16:6810', 'nonce': 4190982331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 4190982331}, {'type': 'v1', 'addr': '172.21.15.16:6814', 'nonce': 4190982331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 4190982331}, {'type': 'v1', 'addr': '172.21.15.16:6822', 'nonce': 4190982331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 4190982331}, {'type': 'v1', 'addr': '172.21.15.16:6818', 'nonce': 4190982331}]}, 'public_addr': '172.21.15.16:6810/4190982331', 'cluster_addr': '172.21.15.16:6814/4190982331', 'heartbeat_back_addr': '172.21.15.16:6822/4190982331', 'heartbeat_front_addr': '172.21.15.16:6818/4190982331', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1686e657-c1f3-499b-8b10-67f428c5c216', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 2266032822}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 2266032822}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 2266032822}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 2266032822}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 2266032822}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 2266032822}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 2266032822}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 2266032822}]}, 'public_addr': '172.21.15.16:6801/2266032822', 'cluster_addr': '172.21.15.16:6803/2266032822', 'heartbeat_back_addr': '172.21.15.16:6807/2266032822', 'heartbeat_front_addr': '172.21.15.16:6805/2266032822', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '730ad37c-1f5f-4d23-a1a7-c75320506665', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6809', 'nonce': 3389562884}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 3389562884}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6813', 'nonce': 3389562884}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 3389562884}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6821', 'nonce': 3389562884}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 3389562884}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6817', 'nonce': 3389562884}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 3389562884}]}, 'public_addr': '172.21.15.16:6811/3389562884', 'cluster_addr': '172.21.15.16:6815/3389562884', 'heartbeat_back_addr': '172.21.15.16:6823/3389562884', 'heartbeat_front_addr': '172.21.15.16:6819/3389562884', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd82c5c44-72fe-4680-9c04-ab4e0775b929', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6810', 'nonce': 1733996396}, {'type': 'v1', 'addr': '172.21.15.174:6811', 'nonce': 1733996396}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6812', 'nonce': 1733996396}, {'type': 'v1', 'addr': '172.21.15.174:6813', 'nonce': 1733996396}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6816', 'nonce': 1733996396}, {'type': 'v1', 'addr': '172.21.15.174:6817', 'nonce': 1733996396}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6814', 'nonce': 1733996396}, {'type': 'v1', 'addr': '172.21.15.174:6815', 'nonce': 1733996396}]}, 'public_addr': '172.21.15.174:6811/1733996396', 'cluster_addr': '172.21.15.174:6813/1733996396', 'heartbeat_back_addr': '172.21.15.174:6817/1733996396', 'heartbeat_front_addr': '172.21.15.174:6815/1733996396', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '41502441-b0d9-460f-92c0-b8b3a915cac5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6826', 'nonce': 2956753716}, {'type': 'v1', 'addr': '172.21.15.174:6827', 'nonce': 2956753716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6828', 'nonce': 2956753716}, {'type': 'v1', 'addr': '172.21.15.174:6829', 'nonce': 2956753716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6832', 'nonce': 2956753716}, {'type': 'v1', 'addr': '172.21.15.174:6833', 'nonce': 2956753716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6830', 'nonce': 2956753716}, {'type': 'v1', 'addr': '172.21.15.174:6831', 'nonce': 2956753716}]}, 'public_addr': '172.21.15.174:6827/2956753716', 'cluster_addr': '172.21.15.174:6829/2956753716', 'heartbeat_back_addr': '172.21.15.174:6833/2956753716', 'heartbeat_front_addr': '172.21.15.174:6831/2956753716', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45cc7521-11db-4496-8d0c-1abc42962810', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6802', 'nonce': 2727832023}, {'type': 'v1', 'addr': '172.21.15.174:6803', 'nonce': 2727832023}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6804', 'nonce': 2727832023}, {'type': 'v1', 'addr': '172.21.15.174:6805', 'nonce': 2727832023}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6808', 'nonce': 2727832023}, {'type': 'v1', 'addr': '172.21.15.174:6809', 'nonce': 2727832023}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6806', 'nonce': 2727832023}, {'type': 'v1', 'addr': '172.21.15.174:6807', 'nonce': 2727832023}]}, 'public_addr': '172.21.15.174:6803/2727832023', 'cluster_addr': '172.21.15.174:6805/2727832023', 'heartbeat_back_addr': '172.21.15.174:6809/2727832023', 'heartbeat_front_addr': '172.21.15.174:6807/2727832023', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1b5d6337-73c3-494b-91cd-fcfa7d307050', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6818', 'nonce': 2338392542}, {'type': 'v1', 'addr': '172.21.15.174:6819', 'nonce': 2338392542}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6820', 'nonce': 2338392542}, {'type': 'v1', 'addr': '172.21.15.174:6821', 'nonce': 2338392542}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6824', 'nonce': 2338392542}, {'type': 'v1', 'addr': '172.21.15.174:6825', 'nonce': 2338392542}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6822', 'nonce': 2338392542}, {'type': 'v1', 'addr': '172.21.15.174:6823', 'nonce': 2338392542}]}, 'public_addr': '172.21.15.174:6819/2338392542', 'cluster_addr': '172.21.15.174:6821/2338392542', 'heartbeat_back_addr': '172.21.15.174:6825/2338392542', 'heartbeat_front_addr': '172.21.15.174:6823/2338392542', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '766efffa-3aff-4aa3-9cf8-88c5c7986d21', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6800', 'nonce': 2621487148}, {'type': 'v1', 'addr': '172.21.15.194:6801', 'nonce': 2621487148}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6802', 'nonce': 2621487148}, {'type': 'v1', 'addr': '172.21.15.194:6803', 'nonce': 2621487148}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6806', 'nonce': 2621487148}, {'type': 'v1', 'addr': '172.21.15.194:6807', 'nonce': 2621487148}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6804', 'nonce': 2621487148}, {'type': 'v1', 'addr': '172.21.15.194:6805', 'nonce': 2621487148}]}, 'public_addr': '172.21.15.194:6801/2621487148', 'cluster_addr': '172.21.15.194:6803/2621487148', 'heartbeat_back_addr': '172.21.15.194:6807/2621487148', 'heartbeat_front_addr': '172.21.15.194:6805/2621487148', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '755cc9aa-2f6f-4d7a-84cf-7feb74d65c48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6816', 'nonce': 367304498}, {'type': 'v1', 'addr': '172.21.15.194:6817', 'nonce': 367304498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6818', 'nonce': 367304498}, {'type': 'v1', 'addr': '172.21.15.194:6819', 'nonce': 367304498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6822', 'nonce': 367304498}, {'type': 'v1', 'addr': '172.21.15.194:6823', 'nonce': 367304498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6820', 'nonce': 367304498}, {'type': 'v1', 'addr': '172.21.15.194:6821', 'nonce': 367304498}]}, 'public_addr': '172.21.15.194:6817/367304498', 'cluster_addr': '172.21.15.194:6819/367304498', 'heartbeat_back_addr': '172.21.15.194:6823/367304498', 'heartbeat_front_addr': '172.21.15.194:6821/367304498', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6824', 'nonce': 3349503344}, {'type': 'v1', 'addr': '172.21.15.194:6825', 'nonce': 3349503344}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6826', 'nonce': 3349503344}, {'type': 'v1', 'addr': '172.21.15.194:6827', 'nonce': 3349503344}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6830', 'nonce': 3349503344}, {'type': 'v1', 'addr': '172.21.15.194:6831', 'nonce': 3349503344}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6828', 'nonce': 3349503344}, {'type': 'v1', 'addr': '172.21.15.194:6829', 'nonce': 3349503344}]}, 'public_addr': '172.21.15.194:6825/3349503344', 'cluster_addr': '172.21.15.194:6827/3349503344', 'heartbeat_back_addr': '172.21.15.194:6831/3349503344', 'heartbeat_front_addr': '172.21.15.194:6829/3349503344', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f71641eb-1331-4b39-9c0e-bf98e91ed295', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6808', 'nonce': 2531896090}, {'type': 'v1', 'addr': '172.21.15.194:6809', 'nonce': 2531896090}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6810', 'nonce': 2531896090}, {'type': 'v1', 'addr': '172.21.15.194:6811', 'nonce': 2531896090}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6814', 'nonce': 2531896090}, {'type': 'v1', 'addr': '172.21.15.194:6815', 'nonce': 2531896090}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6812', 'nonce': 2531896090}, {'type': 'v1', 'addr': '172.21.15.194:6813', 'nonce': 2531896090}]}, 'public_addr': '172.21.15.194:6809/2531896090', 'cluster_addr': '172.21.15.194:6811/2531896090', 'heartbeat_back_addr': '172.21.15.194:6815/2531896090', 'heartbeat_front_addr': '172.21.15.194:6813/2531896090', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:09:01.680673+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T04:08:58.907614+0000', 'dead_epoch': 55}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:08:47.992191+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:08:48.214554+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:07:46.162300+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:08:59.631647+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:08:58.489685+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:09:08.502081+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:08:48.771857+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:09:06.156463+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:08:56.419990+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T04:09:00.745572+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.d', 'osds': [7, 8, 4, 11, 9, 0, 1, 5]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'lrcprofile': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'k': '4', 'l': '3', 'm': '2', 'name': 'lrcprofile', 'plugin': 'lrc'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 22, 'length': 1}, {'begin': 27, 'length': 1}, {'begin': 31, 'length': 1}, {'begin': 33, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 3, 'snaps': [{'begin': 30, 'length': 1}, {'begin': 32, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T04:09:11.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:09:11.028 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:11.028 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:11.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T04:09:11.029 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-09-16T04:09:11.024+0000 7f300eac4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33131) UID: 0 2024-09-16T04:09:11.127 INFO:tasks.ceph.osd.9.smithi194.stderr:2024-09-16T04:09:11.126+0000 7eff4bf54640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33291) UID: 0 2024-09-16T04:09:11.202 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:11.202 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:11.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T04:09:11.259 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:11.257+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:11.323 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:11.323 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:11.329 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:11.328+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:11.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T04:09:11.362 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:11.363 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:09:11.372 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:11.373 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:09:11.376 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-16T04:09:10.684483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":133136947,"num_objects":104,"num_object_clones":53,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":11,"num_read":381,"num_read_kb":202334,"num_write":793,"num_write_kb":122684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":45367315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1655,"ondisk_log_size":1655,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":226,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1038090240,"kb_used":736820,"kb_used_data":264608,"kb_used_omap":17,"kb_used_meta":471918,"kb_avail":1037353420,"statfs":{"total":1063004405760,"available":1062249902080,"internally_reserved":0,"allocated":270958592,"data_stored":380126379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17545,"internal_metadata":483244919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6717440,"num_objects":4,"num_object_clones":3,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":3,"num_read":16,"num_read_kb":18487,"num_write":34,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000833"},"pg_stats":[{"pgid":"3.19","version":"93'57","reported_seq":238,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444289+0000","last_change":"2024-09-16T04:09:08.458196+0000","last_active":"2024-09-16T04:09:09.444289+0000","last_peered":"2024-09-16T04:09:09.444289+0000","last_clean":"2024-09-16T04:09:09.444289+0000","last_became_active":"2024-09-16T04:08:32.459247+0000","last_became_peered":"2024-09-16T04:08:32.459247+0000","last_unstale":"2024-09-16T04:09:09.444289+0000","last_undegraded":"2024-09-16T04:09:09.444289+0000","last_fullsized":"2024-09-16T04:09:09.444289+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'39","last_scrub_stamp":"2024-09-16T04:08:34.300548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.300548+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:59.008546+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0.013408491999999999,"stat_sum":{"num_bytes":1589248,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":3296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4,10,6,7,11,2],"acting":[5,3,4,10,6,7,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.18","version":"92'43","reported_seq":276,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.859547+0000","last_change":"2024-09-16T04:09:08.446539+0000","last_active":"2024-09-16T04:09:09.859547+0000","last_peered":"2024-09-16T04:09:09.859547+0000","last_clean":"2024-09-16T04:09:09.859547+0000","last_became_active":"2024-09-16T04:08:32.465375+0000","last_became_peered":"2024-09-16T04:08:32.465375+0000","last_unstale":"2024-09-16T04:09:09.859547+0000","last_undegraded":"2024-09-16T04:09:09.859547+0000","last_fullsized":"2024-09-16T04:09:09.859547+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'34","last_scrub_stamp":"2024-09-16T04:08:34.203976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.203976+0000","objects_scrubbed":1,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:41.669239+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0.0012341229999999999,"stat_sum":{"num_bytes":278528,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,0,2,8,7,4,5],"acting":[3,9,0,2,8,7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.17","version":"89'77","reported_seq":319,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.449972+0000","last_change":"2024-09-16T04:09:06.450487+0000","last_active":"2024-09-16T04:09:06.449972+0000","last_peered":"2024-09-16T04:09:06.449972+0000","last_clean":"2024-09-16T04:09:06.449972+0000","last_became_active":"2024-09-16T04:08:32.454906+0000","last_became_peered":"2024-09-16T04:08:32.454906+0000","last_unstale":"2024-09-16T04:09:06.449972+0000","last_undegraded":"2024-09-16T04:09:06.449972+0000","last_fullsized":"2024-09-16T04:09:06.449972+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'54","last_scrub_stamp":"2024-09-16T04:08:33.810836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:33.810836+0000","objects_scrubbed":1,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:43.204776+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0.00167925,"stat_sum":{"num_bytes":5537792,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":8,"num_read_kb":4530,"num_write":31,"num_write_kb":3920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,9,3,11,6,2,4],"acting":[0,5,9,3,11,6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"2"}]},{"pgid":"3.16","version":"93'35","reported_seq":298,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444729+0000","last_change":"2024-09-16T04:09:08.472358+0000","last_active":"2024-09-16T04:09:09.444729+0000","last_peered":"2024-09-16T04:09:09.444729+0000","last_clean":"2024-09-16T04:09:09.444729+0000","last_became_active":"2024-09-16T04:08:32.461317+0000","last_became_peered":"2024-09-16T04:08:32.461317+0000","last_unstale":"2024-09-16T04:09:09.444729+0000","last_undegraded":"2024-09-16T04:09:09.444729+0000","last_fullsized":"2024-09-16T04:09:09.444729+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'17","last_scrub_stamp":"2024-09-16T04:08:35.188467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:35.188467+0000","objects_scrubbed":1,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:38.609723+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.027370742,"stat_sum":{"num_bytes":3358720,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":2032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4,3,8,2,10,6],"acting":[5,7,4,3,8,2,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.15","version":"93'87","reported_seq":312,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.464036+0000","last_change":"2024-09-16T04:09:08.464036+0000","last_active":"2024-09-16T04:09:08.464036+0000","last_peered":"2024-09-16T04:09:08.464036+0000","last_clean":"2024-09-16T04:09:08.464036+0000","last_became_active":"2024-09-16T04:08:24.489024+0000","last_became_peered":"2024-09-16T04:08:24.489024+0000","last_unstale":"2024-09-16T04:09:08.464036+0000","last_undegraded":"2024-09-16T04:09:08.464036+0000","last_fullsized":"2024-09-16T04:09:08.464036+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'47","last_scrub_stamp":"2024-09-16T04:08:30.684239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:30.684239+0000","objects_scrubbed":4,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:54.534682+0000","scrub_duration":5,"objects_trimmed":1,"snaptrim_duration":0.018539248000000001,"stat_sum":{"num_bytes":8880128,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":3,"num_read_kb":2,"num_write":53,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,2,6,0,9],"acting":[8,11,7,4,2,6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.14","version":"93'57","reported_seq":319,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.459436+0000","last_change":"2024-09-16T04:09:08.460292+0000","last_active":"2024-09-16T04:09:08.459436+0000","last_peered":"2024-09-16T04:09:08.459436+0000","last_clean":"2024-09-16T04:09:08.459436+0000","last_became_active":"2024-09-16T04:08:24.493499+0000","last_became_peered":"2024-09-16T04:08:24.493499+0000","last_unstale":"2024-09-16T04:09:08.459436+0000","last_undegraded":"2024-09-16T04:09:08.459436+0000","last_fullsized":"2024-09-16T04:09:08.459436+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'24","last_scrub_stamp":"2024-09-16T04:08:31.075306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:31.075306+0000","objects_scrubbed":4,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:44.821039+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0079381169999999997,"stat_sum":{"num_bytes":6946816,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":25,"num_write_kb":4960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,0,11,7,6,2,5],"acting":[4,8,0,11,7,6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.7","version":"0'0","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.443616+0000","last_change":"2024-09-16T04:09:08.905295+0000","last_active":"2024-09-16T04:09:09.443616+0000","last_peered":"2024-09-16T04:09:09.443616+0000","last_clean":"2024-09-16T04:09:09.443616+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:09:09.443616+0000","last_undegraded":"2024-09-16T04:09:09.443616+0000","last_fullsized":"2024-09-16T04:09:09.443616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:08.905173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:08.905173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:32.060373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"89'42","reported_seq":382,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.448539+0000","last_change":"2024-09-16T04:09:06.450413+0000","last_active":"2024-09-16T04:09:06.448539+0000","last_peered":"2024-09-16T04:09:06.448539+0000","last_clean":"2024-09-16T04:09:06.448539+0000","last_became_active":"2024-09-16T04:08:32.459144+0000","last_became_peered":"2024-09-16T04:08:32.459144+0000","last_unstale":"2024-09-16T04:09:06.448539+0000","last_undegraded":"2024-09-16T04:09:06.448539+0000","last_fullsized":"2024-09-16T04:09:06.448539+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'17","last_scrub_stamp":"2024-09-16T04:08:24.393651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:24.393651+0000","objects_scrubbed":1,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:50.352620+0000","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0.0022062079999999999,"stat_sum":{"num_bytes":3555328,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":3472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4,7,11,10,9,5],"acting":[0,8,4,7,11,10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.445249+0000","last_change":"2024-09-16T04:08:32.448925+0000","last_active":"2024-09-16T04:09:08.445249+0000","last_peered":"2024-09-16T04:09:08.445249+0000","last_clean":"2024-09-16T04:09:08.445249+0000","last_became_active":"2024-09-16T04:08:32.448596+0000","last_became_peered":"2024-09-16T04:08:32.448596+0000","last_unstale":"2024-09-16T04:09:08.445249+0000","last_undegraded":"2024-09-16T04:09:08.445249+0000","last_fullsized":"2024-09-16T04:09:08.445249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:09.738883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"93'99","reported_seq":483,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.859732+0000","last_change":"2024-09-16T04:09:08.462408+0000","last_active":"2024-09-16T04:09:09.859732+0000","last_peered":"2024-09-16T04:09:09.859732+0000","last_clean":"2024-09-16T04:09:09.859732+0000","last_became_active":"2024-09-16T04:08:32.458838+0000","last_became_peered":"2024-09-16T04:08:32.458838+0000","last_unstale":"2024-09-16T04:09:09.859732+0000","last_undegraded":"2024-09-16T04:09:09.859732+0000","last_fullsized":"2024-09-16T04:09:09.859732+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'58","last_scrub_stamp":"2024-09-16T04:08:23.959039+0000","last_deep_scrub":"48'58","last_deep_scrub_stamp":"2024-09-16T04:08:23.959039+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.959039+0000","objects_scrubbed":4,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:43.379148+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.01542806,"stat_sum":{"num_bytes":8028160,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":17,"num_read_kb":17484,"num_write":16,"num_write_kb":7840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2883584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7,0,10,9,4,5],"acting":[3,11,7,0,10,9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.445637+0000","last_change":"2024-09-16T04:07:52.939889+0000","last_active":"2024-09-16T04:09:08.445637+0000","last_peered":"2024-09-16T04:09:08.445637+0000","last_clean":"2024-09-16T04:09:08.445637+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:09:08.445637+0000","last_undegraded":"2024-09-16T04:09:08.445637+0000","last_fullsized":"2024-09-16T04:09:08.445637+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:16.774685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'41","reported_seq":241,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.456828+0000","last_change":"2024-09-16T04:09:08.456828+0000","last_active":"2024-09-16T04:09:08.456828+0000","last_peered":"2024-09-16T04:09:08.456828+0000","last_clean":"2024-09-16T04:09:08.456828+0000","last_became_active":"2024-09-16T04:08:32.459250+0000","last_became_peered":"2024-09-16T04:08:32.459250+0000","last_unstale":"2024-09-16T04:09:08.456828+0000","last_undegraded":"2024-09-16T04:09:08.456828+0000","last_fullsized":"2024-09-16T04:09:08.456828+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'18","last_scrub_stamp":"2024-09-16T04:08:35.403029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:35.403029+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:40.727110+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.001614,"stat_sum":{"num_bytes":4063232,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":11907,"num_write":11,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,9,5,7,2,6,0],"acting":[10,11,9,5,7,2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.4","version":"0'0","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.447990+0000","last_change":"2024-09-16T04:08:54.852555+0000","last_active":"2024-09-16T04:09:06.447990+0000","last_peered":"2024-09-16T04:09:06.447990+0000","last_clean":"2024-09-16T04:09:06.447990+0000","last_became_active":"2024-09-16T04:08:32.451119+0000","last_became_peered":"2024-09-16T04:08:32.451119+0000","last_unstale":"2024-09-16T04:09:06.447990+0000","last_undegraded":"2024-09-16T04:09:06.447990+0000","last_fullsized":"2024-09-16T04:09:06.447990+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:54.852438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:54.852438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:58.547468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"93'60","reported_seq":425,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444720+0000","last_change":"2024-09-16T04:09:08.460861+0000","last_active":"2024-09-16T04:09:09.444720+0000","last_peered":"2024-09-16T04:09:09.444720+0000","last_clean":"2024-09-16T04:09:09.444720+0000","last_became_active":"2024-09-16T04:08:32.460408+0000","last_became_peered":"2024-09-16T04:08:32.460408+0000","last_unstale":"2024-09-16T04:09:09.444720+0000","last_undegraded":"2024-09-16T04:09:09.444720+0000","last_fullsized":"2024-09-16T04:09:09.444720+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'43","last_scrub_stamp":"2024-09-16T04:08:23.237695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.237695+0000","objects_scrubbed":1,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:50.507139+0000","scrub_duration":4,"objects_trimmed":1,"snaptrim_duration":0.016047895,"stat_sum":{"num_bytes":6307840,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":7,"num_write_kb":2336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3915776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6,7,8,11,10],"acting":[5,3,2,6,7,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.3","version":"93'28","reported_seq":385,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.506525+0000","last_change":"2024-09-16T04:09:08.454625+0000","last_active":"2024-09-16T04:09:08.506525+0000","last_peered":"2024-09-16T04:09:08.506525+0000","last_clean":"2024-09-16T04:09:08.506525+0000","last_became_active":"2024-09-16T04:08:22.445499+0000","last_became_peered":"2024-09-16T04:08:22.445499+0000","last_unstale":"2024-09-16T04:09:08.506525+0000","last_undegraded":"2024-09-16T04:09:08.506525+0000","last_fullsized":"2024-09-16T04:09:08.506525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'16","last_scrub_stamp":"2024-09-16T04:08:23.064773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.064773+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:50.972621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0019763039999999999,"stat_sum":{"num_bytes":5079040,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,6,11,5,9,10],"acting":[4,3,0,6,11,5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.2","version":"84'3","reported_seq":180,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444529+0000","last_change":"2024-09-16T04:09:00.126377+0000","last_active":"2024-09-16T04:09:09.444529+0000","last_peered":"2024-09-16T04:09:09.444529+0000","last_clean":"2024-09-16T04:09:09.444529+0000","last_became_active":"2024-09-16T04:08:32.457934+0000","last_became_peered":"2024-09-16T04:08:32.457934+0000","last_unstale":"2024-09-16T04:09:09.444529+0000","last_undegraded":"2024-09-16T04:09:09.444529+0000","last_fullsized":"2024-09-16T04:09:09.444529+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3","last_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_deep_scrub":"84'3","last_deep_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_clean_scrub_stamp":"2024-09-16T04:09:00.126257+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:01.779885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"93'12","reported_seq":370,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.457364+0000","last_change":"2024-09-16T04:09:08.458178+0000","last_active":"2024-09-16T04:09:08.457364+0000","last_peered":"2024-09-16T04:09:08.457364+0000","last_clean":"2024-09-16T04:09:08.457364+0000","last_became_active":"2024-09-16T04:08:32.459208+0000","last_became_peered":"2024-09-16T04:08:32.459208+0000","last_unstale":"2024-09-16T04:09:08.457364+0000","last_undegraded":"2024-09-16T04:09:08.457364+0000","last_fullsized":"2024-09-16T04:09:08.457364+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-09-16T04:08:23.448368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.448368+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:53.308793+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0.011966566,"stat_sum":{"num_bytes":196608,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":1154,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6,3,7,8,0,5],"acting":[11,9,6,3,7,8,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.1","version":"0'0","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.540795+0000","last_change":"2024-09-16T04:07:53.018517+0000","last_active":"2024-09-16T04:09:09.540795+0000","last_peered":"2024-09-16T04:09:09.540795+0000","last_clean":"2024-09-16T04:09:09.540795+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:09:09.540795+0000","last_undegraded":"2024-09-16T04:09:09.540795+0000","last_fullsized":"2024-09-16T04:09:09.540795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'34","reported_seq":374,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.448463+0000","last_change":"2024-09-16T04:09:06.449833+0000","last_active":"2024-09-16T04:09:06.448463+0000","last_peered":"2024-09-16T04:09:06.448463+0000","last_clean":"2024-09-16T04:09:06.448463+0000","last_became_active":"2024-09-16T04:08:32.459260+0000","last_became_peered":"2024-09-16T04:08:32.459260+0000","last_unstale":"2024-09-16T04:09:06.448463+0000","last_undegraded":"2024-09-16T04:09:06.448463+0000","last_fullsized":"2024-09-16T04:09:06.448463+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'15","last_scrub_stamp":"2024-09-16T04:08:22.680404+0000","last_deep_scrub":"45'15","last_deep_scrub_stamp":"2024-09-16T04:08:22.680404+0000","last_clean_scrub_stamp":"2024-09-16T04:08:22.680404+0000","objects_scrubbed":1,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:41.929432+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.002048542,"stat_sum":{"num_bytes":1785856,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":28,"num_write_kb":3760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10,11,5,9,6,7],"acting":[0,4,10,11,5,9,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.450229+0000","last_change":"2024-09-16T04:08:58.876711+0000","last_active":"2024-09-16T04:09:08.450229+0000","last_peered":"2024-09-16T04:09:08.450229+0000","last_clean":"2024-09-16T04:09:08.450229+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:09:08.450229+0000","last_undegraded":"2024-09-16T04:09:08.450229+0000","last_fullsized":"2024-09-16T04:09:08.450229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_clean_scrub_stamp":"2024-09-16T04:08:58.876583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:28.287717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'119","reported_seq":511,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.859542+0000","last_change":"2024-09-16T04:09:08.449008+0000","last_active":"2024-09-16T04:09:09.859542+0000","last_peered":"2024-09-16T04:09:09.859542+0000","last_clean":"2024-09-16T04:09:09.859542+0000","last_became_active":"2024-09-16T04:08:32.466786+0000","last_became_peered":"2024-09-16T04:08:32.466786+0000","last_unstale":"2024-09-16T04:09:09.859542+0000","last_undegraded":"2024-09-16T04:09:09.859542+0000","last_fullsized":"2024-09-16T04:09:09.859542+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'52","last_scrub_stamp":"2024-09-16T04:08:23.356294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.356294+0000","objects_scrubbed":3,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:39.907457+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0.0023891390000000002,"stat_sum":{"num_bytes":9109504,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":25162,"num_write":62,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":6144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6,4,5,7,8,10],"acting":[3,9,6,4,5,7,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.3","version":"21'1","reported_seq":170,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444559+0000","last_change":"2024-09-16T04:08:59.147922+0000","last_active":"2024-09-16T04:09:09.444559+0000","last_peered":"2024-09-16T04:09:09.444559+0000","last_clean":"2024-09-16T04:09:09.444559+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:09:09.444559+0000","last_undegraded":"2024-09-16T04:09:09.444559+0000","last_fullsized":"2024-09-16T04:09:09.444559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-09-16T04:08:59.147797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:59.147797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:25.871032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":335,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.455690+0000","last_change":"2024-09-16T04:08:55.861559+0000","last_active":"2024-09-16T04:09:08.455690+0000","last_peered":"2024-09-16T04:09:08.455690+0000","last_clean":"2024-09-16T04:09:08.455690+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:09:08.455690+0000","last_undegraded":"2024-09-16T04:09:08.455690+0000","last_fullsized":"2024-09-16T04:09:08.455690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'137","last_scrub_stamp":"2024-09-16T04:08:55.861468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:08:55.861468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:57.766582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"92'66","reported_seq":441,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.446601+0000","last_change":"2024-09-16T04:09:08.447541+0000","last_active":"2024-09-16T04:09:08.446601+0000","last_peered":"2024-09-16T04:09:08.446601+0000","last_clean":"2024-09-16T04:09:08.446601+0000","last_became_active":"2024-09-16T04:08:32.466625+0000","last_became_peered":"2024-09-16T04:08:32.466625+0000","last_unstale":"2024-09-16T04:09:08.446601+0000","last_undegraded":"2024-09-16T04:09:08.446601+0000","last_fullsized":"2024-09-16T04:09:08.446601+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'38","last_scrub_stamp":"2024-09-16T04:08:24.203960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:24.203960+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:44.293708+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0.0012348089999999999,"stat_sum":{"num_bytes":5554176,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":26212,"num_write":10,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5390336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2,4,3,5,10,8],"acting":[11,7,2,4,3,5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.9","version":"93'119","reported_seq":489,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.459597+0000","last_change":"2024-09-16T04:09:08.462883+0000","last_active":"2024-09-16T04:09:08.459597+0000","last_peered":"2024-09-16T04:09:08.459597+0000","last_clean":"2024-09-16T04:09:08.459597+0000","last_became_active":"2024-09-16T04:08:32.454824+0000","last_became_peered":"2024-09-16T04:08:32.454824+0000","last_unstale":"2024-09-16T04:09:08.459597+0000","last_undegraded":"2024-09-16T04:09:08.459597+0000","last_fullsized":"2024-09-16T04:09:08.459597+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'39","last_scrub_stamp":"2024-09-16T04:08:23.214261+0000","last_deep_scrub":"45'39","last_deep_scrub_stamp":"2024-09-16T04:08:23.214261+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.214261+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:32.700919+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0.017477276,"stat_sum":{"num_bytes":12943360,"num_objects":9,"num_object_clones":6,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":14,"num_read_kb":21317,"num_write":24,"num_write_kb":6304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11059200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,9,5,6,4,10],"acting":[8,11,7,9,5,6,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.a","version":"93'69","reported_seq":418,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.443753+0000","last_change":"2024-09-16T04:09:08.494544+0000","last_active":"2024-09-16T04:09:09.443753+0000","last_peered":"2024-09-16T04:09:09.443753+0000","last_clean":"2024-09-16T04:09:09.443753+0000","last_became_active":"2024-09-16T04:08:32.458689+0000","last_became_peered":"2024-09-16T04:08:32.458689+0000","last_unstale":"2024-09-16T04:09:09.443753+0000","last_undegraded":"2024-09-16T04:09:09.443753+0000","last_fullsized":"2024-09-16T04:09:09.443753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:28.335645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.048377525999999997,"stat_sum":{"num_bytes":9338880,"num_objects":6,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":16,"num_read_kb":5,"num_write":24,"num_write_kb":13584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4620288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5,10,8,9,3,7],"acting":[6,4,5,10,8,9,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.b","version":"93'102","reported_seq":487,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.859705+0000","last_change":"2024-09-16T04:09:08.450149+0000","last_active":"2024-09-16T04:09:09.859705+0000","last_peered":"2024-09-16T04:09:09.859705+0000","last_clean":"2024-09-16T04:09:09.859705+0000","last_became_active":"2024-09-16T04:08:32.458688+0000","last_became_peered":"2024-09-16T04:08:32.458688+0000","last_unstale":"2024-09-16T04:09:09.859705+0000","last_undegraded":"2024-09-16T04:09:09.859705+0000","last_fullsized":"2024-09-16T04:09:09.859705+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:31.699655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027560810000000001,"stat_sum":{"num_bytes":12632064,"num_objects":8,"num_object_clones":4,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":56481,"num_write":72,"num_write_kb":13632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3620864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4,8,11,7,6,9],"acting":[3,0,4,8,11,7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.c","version":"93'95","reported_seq":445,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444870+0000","last_change":"2024-09-16T04:09:08.478728+0000","last_active":"2024-09-16T04:09:09.444870+0000","last_peered":"2024-09-16T04:09:09.444870+0000","last_clean":"2024-09-16T04:09:09.444870+0000","last_became_active":"2024-09-16T04:08:03.222541+0000","last_became_peered":"2024-09-16T04:08:03.222541+0000","last_unstale":"2024-09-16T04:09:09.444870+0000","last_undegraded":"2024-09-16T04:09:09.444870+0000","last_fullsized":"2024-09-16T04:09:09.444870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:31.031193+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.033256309999999997,"stat_sum":{"num_bytes":6832128,"num_objects":9,"num_object_clones":4,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":46,"num_read_kb":7125,"num_write":97,"num_write_kb":10336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6,10,9,2,0,11],"acting":[5,3,6,10,9,2,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.d","version":"92'46","reported_seq":434,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.457832+0000","last_change":"2024-09-16T04:09:08.458188+0000","last_active":"2024-09-16T04:09:08.457832+0000","last_peered":"2024-09-16T04:09:08.457832+0000","last_clean":"2024-09-16T04:09:08.457832+0000","last_became_active":"2024-09-16T04:08:32.461735+0000","last_became_peered":"2024-09-16T04:08:32.461735+0000","last_unstale":"2024-09-16T04:09:08.457832+0000","last_undegraded":"2024-09-16T04:09:08.457832+0000","last_fullsized":"2024-09-16T04:09:08.457832+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:24.263946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073052399999999997,"stat_sum":{"num_bytes":1818624,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":243,"num_write":40,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2473984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9,6,11,3,4,0],"acting":[7,5,9,6,11,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.e","version":"69'19","reported_seq":360,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.446189+0000","last_change":"2024-09-16T04:09:08.446541+0000","last_active":"2024-09-16T04:09:08.446189+0000","last_peered":"2024-09-16T04:09:08.446189+0000","last_clean":"2024-09-16T04:09:08.446189+0000","last_became_active":"2024-09-16T04:08:32.457538+0000","last_became_peered":"2024-09-16T04:08:32.457538+0000","last_unstale":"2024-09-16T04:09:08.446189+0000","last_undegraded":"2024-09-16T04:09:08.446189+0000","last_fullsized":"2024-09-16T04:09:08.446189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:26.691248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053797600000000001,"stat_sum":{"num_bytes":3817472,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":22372,"num_write":7,"num_write_kb":4208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6,4,0,9,3,10],"acting":[11,8,6,4,0,9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.f","version":"92'18","reported_seq":348,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.450241+0000","last_change":"2024-09-16T04:09:08.451194+0000","last_active":"2024-09-16T04:09:08.450241+0000","last_peered":"2024-09-16T04:09:08.450241+0000","last_clean":"2024-09-16T04:09:08.450241+0000","last_became_active":"2024-09-16T04:08:03.221547+0000","last_became_peered":"2024-09-16T04:08:03.221547+0000","last_unstale":"2024-09-16T04:09:08.450241+0000","last_undegraded":"2024-09-16T04:09:08.450241+0000","last_fullsized":"2024-09-16T04:09:08.450241+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'16","last_scrub_stamp":"2024-09-16T04:09:06.936150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:09:06.936150+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:17.559188+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.001125734,"stat_sum":{"num_bytes":1474560,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8,2,9,10,0,4],"acting":[7,11,8,2,9,10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.10","version":"30'2","reported_seq":254,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.443229+0000","last_change":"2024-09-16T04:09:08.447028+0000","last_active":"2024-09-16T04:09:09.443229+0000","last_peered":"2024-09-16T04:09:09.443229+0000","last_clean":"2024-09-16T04:09:09.443229+0000","last_became_active":"2024-09-16T04:08:32.459694+0000","last_became_peered":"2024-09-16T04:08:32.459694+0000","last_unstale":"2024-09-16T04:09:09.443229+0000","last_undegraded":"2024-09-16T04:09:09.443229+0000","last_fullsized":"2024-09-16T04:09:09.443229+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'2","last_scrub_stamp":"2024-09-16T04:08:34.079799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.079799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:46.373607+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0012269049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5,3,0,2,10,7],"acting":[6,9,5,3,0,2,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.11","version":"80'32","reported_seq":283,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.454746+0000","last_change":"2024-09-16T04:09:08.456883+0000","last_active":"2024-09-16T04:09:08.454746+0000","last_peered":"2024-09-16T04:09:08.454746+0000","last_clean":"2024-09-16T04:09:08.454746+0000","last_became_active":"2024-09-16T04:08:32.455596+0000","last_became_peered":"2024-09-16T04:08:32.455596+0000","last_unstale":"2024-09-16T04:09:08.454746+0000","last_undegraded":"2024-09-16T04:09:08.454746+0000","last_fullsized":"2024-09-16T04:09:08.454746+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'15","last_scrub_stamp":"2024-09-16T04:08:34.403063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.403063+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:40.573546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023664620000000002,"stat_sum":{"num_bytes":2146304,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6364,"num_write":26,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6,0,3,9,8,11],"acting":[10,4,6,0,3,9,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.12","version":"89'119","reported_seq":397,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.450622+0000","last_change":"2024-09-16T04:09:06.451427+0000","last_active":"2024-09-16T04:09:06.450622+0000","last_peered":"2024-09-16T04:09:06.450622+0000","last_clean":"2024-09-16T04:09:06.450622+0000","last_became_active":"2024-09-16T04:08:32.460980+0000","last_became_peered":"2024-09-16T04:08:32.460980+0000","last_unstale":"2024-09-16T04:09:06.450622+0000","last_undegraded":"2024-09-16T04:09:06.450622+0000","last_fullsized":"2024-09-16T04:09:06.450622+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'62","last_scrub_stamp":"2024-09-16T04:08:35.178702+0000","last_deep_scrub":"57'62","last_deep_scrub_stamp":"2024-09-16T04:08:35.178702+0000","last_clean_scrub_stamp":"2024-09-16T04:08:35.178702+0000","objects_scrubbed":6,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:52.750345+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.002257359,"stat_sum":{"num_bytes":9420800,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":10,"num_read_kb":1857,"num_write":16,"num_write_kb":5024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,8,2,5,6,11],"acting":[0,7,3,8,2,5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"2"}]},{"pgid":"3.13","version":"93'36","reported_seq":286,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.460931+0000","last_change":"2024-09-16T04:09:08.462370+0000","last_active":"2024-09-16T04:09:08.460931+0000","last_peered":"2024-09-16T04:09:08.460931+0000","last_clean":"2024-09-16T04:09:08.460931+0000","last_became_active":"2024-09-16T04:08:32.463080+0000","last_became_peered":"2024-09-16T04:08:32.463080+0000","last_unstale":"2024-09-16T04:09:08.460931+0000","last_undegraded":"2024-09-16T04:09:08.460931+0000","last_fullsized":"2024-09-16T04:09:08.460931+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"37'16","last_scrub_stamp":"2024-09-16T04:08:34.081657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.081657+0000","objects_scrubbed":1,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:01.908123+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.012386222000000001,"stat_sum":{"num_bytes":1851392,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4,2,6,3,9,8],"acting":[7,5,4,2,6,3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":132546560,"num_objects":100,"num_object_clones":53,"num_object_copies":800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":11,"num_read":255,"num_read_kb":202225,"num_write":610,"num_write_kb":119744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":45367296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270499840,"data_stored":377106432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":585,"internal_metadata":0},"log_size":1514,"ondisk_log_size":1514,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378643,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77600,"kb_used_data":28364,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94294240,"statfs":{"total":96636764160,"available":96557301760,"internally_reserved":0,"allocated":29044736,"data_stored":38233732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378642,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79312,"kb_used_data":25908,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94292528,"statfs":{"total":96636764160,"available":96555548672,"internally_reserved":0,"allocated":26529792,"data_stored":36288132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378642,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75668,"kb_used_data":26424,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94296172,"statfs":{"total":96636764160,"available":96559280128,"internally_reserved":0,"allocated":27058176,"data_stored":36292209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378645,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77204,"kb_used_data":23800,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94294636,"statfs":{"total":96636764160,"available":96557707264,"internally_reserved":0,"allocated":24371200,"data_stored":31786609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51628,"kb_used_data":16280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320212,"statfs":{"total":96636764160,"available":96583897088,"internally_reserved":0,"allocated":16670720,"data_stored":22951537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411346,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79932,"kb_used_data":26528,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94291908,"statfs":{"total":96636764160,"available":96554913792,"internally_reserved":0,"allocated":27164672,"data_stored":42420035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411348,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52488,"kb_used_data":21296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319352,"statfs":{"total":96636764160,"available":96583016448,"internally_reserved":0,"allocated":21807104,"data_stored":29578865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411346,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72992,"kb_used_data":23752,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94298848,"statfs":{"total":96636764160,"available":96562020352,"internally_reserved":0,"allocated":24322048,"data_stored":34497827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411346,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55620,"kb_used_data":24428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316220,"statfs":{"total":96636764160,"available":96579809280,"internally_reserved":0,"allocated":25014272,"data_stored":33543793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411348,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58896,"kb_used_data":27700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94312944,"statfs":{"total":96636764160,"available":96576454656,"internally_reserved":0,"allocated":28364800,"data_stored":42628931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":0,"seq":326417514500,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444051,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55480,"kb_used_data":20128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316360,"statfs":{"total":96636764160,"available":96579952640,"internally_reserved":0,"allocated":20611072,"data_stored":31904709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19554304,"data_stored":31244288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10178560,"data_stored":12312576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":585,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15589376,"data_stored":22274048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23289856,"data_stored":31109120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25976832,"data_stored":35614720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25444352,"data_stored":35610624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27959296,"data_stored":37556224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26763264,"data_stored":38789120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":23932928,"data_stored":32866304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24387584,"data_stored":32493568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":20725760,"data_stored":28901376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26697728,"data_stored":38334464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:09:11.378 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-16T04:09:11.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-16T04:09:11.385 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-16T04:09:10.684483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":133136947,"num_objects":104,"num_object_clones":53,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":11,"num_read":381,"num_read_kb":202334,"num_write":793,"num_write_kb":122684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":45367315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1655,"ondisk_log_size":1655,"up":226,"acting":226,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":226,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1038090240,"kb_used":736820,"kb_used_data":264608,"kb_used_omap":17,"kb_used_meta":471918,"kb_avail":1037353420,"statfs":{"total":1063004405760,"available":1062249902080,"internally_reserved":0,"allocated":270958592,"data_stored":380126379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17545,"internal_metadata":483244919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6717440,"num_objects":4,"num_object_clones":3,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":3,"num_read":16,"num_read_kb":18487,"num_write":34,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000833"},"pg_stats":[{"pgid":"3.19","version":"93'57","reported_seq":238,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444289+0000","last_change":"2024-09-16T04:09:08.458196+0000","last_active":"2024-09-16T04:09:09.444289+0000","last_peered":"2024-09-16T04:09:09.444289+0000","last_clean":"2024-09-16T04:09:09.444289+0000","last_became_active":"2024-09-16T04:08:32.459247+0000","last_became_peered":"2024-09-16T04:08:32.459247+0000","last_unstale":"2024-09-16T04:09:09.444289+0000","last_undegraded":"2024-09-16T04:09:09.444289+0000","last_fullsized":"2024-09-16T04:09:09.444289+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'39","last_scrub_stamp":"2024-09-16T04:08:34.300548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.300548+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:59.008546+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0.013408491999999999,"stat_sum":{"num_bytes":1589248,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":3296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4,10,6,7,11,2],"acting":[5,3,4,10,6,7,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.18","version":"92'43","reported_seq":276,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.859547+0000","last_change":"2024-09-16T04:09:08.446539+0000","last_active":"2024-09-16T04:09:09.859547+0000","last_peered":"2024-09-16T04:09:09.859547+0000","last_clean":"2024-09-16T04:09:09.859547+0000","last_became_active":"2024-09-16T04:08:32.465375+0000","last_became_peered":"2024-09-16T04:08:32.465375+0000","last_unstale":"2024-09-16T04:09:09.859547+0000","last_undegraded":"2024-09-16T04:09:09.859547+0000","last_fullsized":"2024-09-16T04:09:09.859547+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'34","last_scrub_stamp":"2024-09-16T04:08:34.203976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.203976+0000","objects_scrubbed":1,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:41.669239+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0.0012341229999999999,"stat_sum":{"num_bytes":278528,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,0,2,8,7,4,5],"acting":[3,9,0,2,8,7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.17","version":"89'77","reported_seq":319,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.449972+0000","last_change":"2024-09-16T04:09:06.450487+0000","last_active":"2024-09-16T04:09:06.449972+0000","last_peered":"2024-09-16T04:09:06.449972+0000","last_clean":"2024-09-16T04:09:06.449972+0000","last_became_active":"2024-09-16T04:08:32.454906+0000","last_became_peered":"2024-09-16T04:08:32.454906+0000","last_unstale":"2024-09-16T04:09:06.449972+0000","last_undegraded":"2024-09-16T04:09:06.449972+0000","last_fullsized":"2024-09-16T04:09:06.449972+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'54","last_scrub_stamp":"2024-09-16T04:08:33.810836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:33.810836+0000","objects_scrubbed":1,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:43.204776+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0.00167925,"stat_sum":{"num_bytes":5537792,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":8,"num_read_kb":4530,"num_write":31,"num_write_kb":3920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,9,3,11,6,2,4],"acting":[0,5,9,3,11,6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"2"}]},{"pgid":"3.16","version":"93'35","reported_seq":298,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444729+0000","last_change":"2024-09-16T04:09:08.472358+0000","last_active":"2024-09-16T04:09:09.444729+0000","last_peered":"2024-09-16T04:09:09.444729+0000","last_clean":"2024-09-16T04:09:09.444729+0000","last_became_active":"2024-09-16T04:08:32.461317+0000","last_became_peered":"2024-09-16T04:08:32.461317+0000","last_unstale":"2024-09-16T04:09:09.444729+0000","last_undegraded":"2024-09-16T04:09:09.444729+0000","last_fullsized":"2024-09-16T04:09:09.444729+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"39'17","last_scrub_stamp":"2024-09-16T04:08:35.188467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:35.188467+0000","objects_scrubbed":1,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:38.609723+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.027370742,"stat_sum":{"num_bytes":3358720,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":2032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4,3,8,2,10,6],"acting":[5,7,4,3,8,2,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.15","version":"93'87","reported_seq":312,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.464036+0000","last_change":"2024-09-16T04:09:08.464036+0000","last_active":"2024-09-16T04:09:08.464036+0000","last_peered":"2024-09-16T04:09:08.464036+0000","last_clean":"2024-09-16T04:09:08.464036+0000","last_became_active":"2024-09-16T04:08:24.489024+0000","last_became_peered":"2024-09-16T04:08:24.489024+0000","last_unstale":"2024-09-16T04:09:08.464036+0000","last_undegraded":"2024-09-16T04:09:08.464036+0000","last_fullsized":"2024-09-16T04:09:08.464036+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'47","last_scrub_stamp":"2024-09-16T04:08:30.684239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:30.684239+0000","objects_scrubbed":4,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:54.534682+0000","scrub_duration":5,"objects_trimmed":1,"snaptrim_duration":0.018539248000000001,"stat_sum":{"num_bytes":8880128,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":3,"num_read_kb":2,"num_write":53,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,4,2,6,0,9],"acting":[8,11,7,4,2,6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.14","version":"93'57","reported_seq":319,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.459436+0000","last_change":"2024-09-16T04:09:08.460292+0000","last_active":"2024-09-16T04:09:08.459436+0000","last_peered":"2024-09-16T04:09:08.459436+0000","last_clean":"2024-09-16T04:09:08.459436+0000","last_became_active":"2024-09-16T04:08:24.493499+0000","last_became_peered":"2024-09-16T04:08:24.493499+0000","last_unstale":"2024-09-16T04:09:08.459436+0000","last_undegraded":"2024-09-16T04:09:08.459436+0000","last_fullsized":"2024-09-16T04:09:08.459436+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'24","last_scrub_stamp":"2024-09-16T04:08:31.075306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:31.075306+0000","objects_scrubbed":4,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:44.821039+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0079381169999999997,"stat_sum":{"num_bytes":6946816,"num_objects":4,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":25,"num_write_kb":4960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,0,11,7,6,2,5],"acting":[4,8,0,11,7,6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.7","version":"0'0","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.443616+0000","last_change":"2024-09-16T04:09:08.905295+0000","last_active":"2024-09-16T04:09:09.443616+0000","last_peered":"2024-09-16T04:09:09.443616+0000","last_clean":"2024-09-16T04:09:09.443616+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:09:09.443616+0000","last_undegraded":"2024-09-16T04:09:09.443616+0000","last_fullsized":"2024-09-16T04:09:09.443616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:08.905173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:08.905173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:32.060373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"89'42","reported_seq":382,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.448539+0000","last_change":"2024-09-16T04:09:06.450413+0000","last_active":"2024-09-16T04:09:06.448539+0000","last_peered":"2024-09-16T04:09:06.448539+0000","last_clean":"2024-09-16T04:09:06.448539+0000","last_became_active":"2024-09-16T04:08:32.459144+0000","last_became_peered":"2024-09-16T04:08:32.459144+0000","last_unstale":"2024-09-16T04:09:06.448539+0000","last_undegraded":"2024-09-16T04:09:06.448539+0000","last_fullsized":"2024-09-16T04:09:06.448539+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'17","last_scrub_stamp":"2024-09-16T04:08:24.393651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:24.393651+0000","objects_scrubbed":1,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:50.352620+0000","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0.0022062079999999999,"stat_sum":{"num_bytes":3555328,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":3472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4,7,11,10,9,5],"acting":[0,8,4,7,11,10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.445249+0000","last_change":"2024-09-16T04:08:32.448925+0000","last_active":"2024-09-16T04:09:08.445249+0000","last_peered":"2024-09-16T04:09:08.445249+0000","last_clean":"2024-09-16T04:09:08.445249+0000","last_became_active":"2024-09-16T04:08:32.448596+0000","last_became_peered":"2024-09-16T04:08:32.448596+0000","last_unstale":"2024-09-16T04:09:08.445249+0000","last_undegraded":"2024-09-16T04:09:08.445249+0000","last_fullsized":"2024-09-16T04:09:08.445249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:09.738883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"93'99","reported_seq":483,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.859732+0000","last_change":"2024-09-16T04:09:08.462408+0000","last_active":"2024-09-16T04:09:09.859732+0000","last_peered":"2024-09-16T04:09:09.859732+0000","last_clean":"2024-09-16T04:09:09.859732+0000","last_became_active":"2024-09-16T04:08:32.458838+0000","last_became_peered":"2024-09-16T04:08:32.458838+0000","last_unstale":"2024-09-16T04:09:09.859732+0000","last_undegraded":"2024-09-16T04:09:09.859732+0000","last_fullsized":"2024-09-16T04:09:09.859732+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'58","last_scrub_stamp":"2024-09-16T04:08:23.959039+0000","last_deep_scrub":"48'58","last_deep_scrub_stamp":"2024-09-16T04:08:23.959039+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.959039+0000","objects_scrubbed":4,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:43.379148+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.01542806,"stat_sum":{"num_bytes":8028160,"num_objects":6,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":17,"num_read_kb":17484,"num_write":16,"num_write_kb":7840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2883584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7,0,10,9,4,5],"acting":[3,11,7,0,10,9,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.445637+0000","last_change":"2024-09-16T04:07:52.939889+0000","last_active":"2024-09-16T04:09:08.445637+0000","last_peered":"2024-09-16T04:09:08.445637+0000","last_clean":"2024-09-16T04:09:08.445637+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:09:08.445637+0000","last_undegraded":"2024-09-16T04:09:08.445637+0000","last_fullsized":"2024-09-16T04:09:08.445637+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:16.774685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'41","reported_seq":241,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.456828+0000","last_change":"2024-09-16T04:09:08.456828+0000","last_active":"2024-09-16T04:09:08.456828+0000","last_peered":"2024-09-16T04:09:08.456828+0000","last_clean":"2024-09-16T04:09:08.456828+0000","last_became_active":"2024-09-16T04:08:32.459250+0000","last_became_peered":"2024-09-16T04:08:32.459250+0000","last_unstale":"2024-09-16T04:09:08.456828+0000","last_undegraded":"2024-09-16T04:09:08.456828+0000","last_fullsized":"2024-09-16T04:09:08.456828+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'18","last_scrub_stamp":"2024-09-16T04:08:35.403029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:35.403029+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:40.727110+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.001614,"stat_sum":{"num_bytes":4063232,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":11907,"num_write":11,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,9,5,7,2,6,0],"acting":[10,11,9,5,7,2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.4","version":"0'0","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.447990+0000","last_change":"2024-09-16T04:08:54.852555+0000","last_active":"2024-09-16T04:09:06.447990+0000","last_peered":"2024-09-16T04:09:06.447990+0000","last_clean":"2024-09-16T04:09:06.447990+0000","last_became_active":"2024-09-16T04:08:32.451119+0000","last_became_peered":"2024-09-16T04:08:32.451119+0000","last_unstale":"2024-09-16T04:09:06.447990+0000","last_undegraded":"2024-09-16T04:09:06.447990+0000","last_fullsized":"2024-09-16T04:09:06.447990+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:54.852438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:54.852438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:58.547468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"93'60","reported_seq":425,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444720+0000","last_change":"2024-09-16T04:09:08.460861+0000","last_active":"2024-09-16T04:09:09.444720+0000","last_peered":"2024-09-16T04:09:09.444720+0000","last_clean":"2024-09-16T04:09:09.444720+0000","last_became_active":"2024-09-16T04:08:32.460408+0000","last_became_peered":"2024-09-16T04:08:32.460408+0000","last_unstale":"2024-09-16T04:09:09.444720+0000","last_undegraded":"2024-09-16T04:09:09.444720+0000","last_fullsized":"2024-09-16T04:09:09.444720+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'43","last_scrub_stamp":"2024-09-16T04:08:23.237695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.237695+0000","objects_scrubbed":1,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:50.507139+0000","scrub_duration":4,"objects_trimmed":1,"snaptrim_duration":0.016047895,"stat_sum":{"num_bytes":6307840,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":7,"num_write_kb":2336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3915776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6,7,8,11,10],"acting":[5,3,2,6,7,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.3","version":"93'28","reported_seq":385,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.506525+0000","last_change":"2024-09-16T04:09:08.454625+0000","last_active":"2024-09-16T04:09:08.506525+0000","last_peered":"2024-09-16T04:09:08.506525+0000","last_clean":"2024-09-16T04:09:08.506525+0000","last_became_active":"2024-09-16T04:08:22.445499+0000","last_became_peered":"2024-09-16T04:08:22.445499+0000","last_unstale":"2024-09-16T04:09:08.506525+0000","last_undegraded":"2024-09-16T04:09:08.506525+0000","last_fullsized":"2024-09-16T04:09:08.506525+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'16","last_scrub_stamp":"2024-09-16T04:08:23.064773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.064773+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:50.972621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0019763039999999999,"stat_sum":{"num_bytes":5079040,"num_objects":4,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,6,11,5,9,10],"acting":[4,3,0,6,11,5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.2","version":"84'3","reported_seq":180,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444529+0000","last_change":"2024-09-16T04:09:00.126377+0000","last_active":"2024-09-16T04:09:09.444529+0000","last_peered":"2024-09-16T04:09:09.444529+0000","last_clean":"2024-09-16T04:09:09.444529+0000","last_became_active":"2024-09-16T04:08:32.457934+0000","last_became_peered":"2024-09-16T04:08:32.457934+0000","last_unstale":"2024-09-16T04:09:09.444529+0000","last_undegraded":"2024-09-16T04:09:09.444529+0000","last_fullsized":"2024-09-16T04:09:09.444529+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3","last_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_deep_scrub":"84'3","last_deep_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_clean_scrub_stamp":"2024-09-16T04:09:00.126257+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:01.779885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"93'12","reported_seq":370,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.457364+0000","last_change":"2024-09-16T04:09:08.458178+0000","last_active":"2024-09-16T04:09:08.457364+0000","last_peered":"2024-09-16T04:09:08.457364+0000","last_clean":"2024-09-16T04:09:08.457364+0000","last_became_active":"2024-09-16T04:08:32.459208+0000","last_became_peered":"2024-09-16T04:08:32.459208+0000","last_unstale":"2024-09-16T04:09:08.457364+0000","last_undegraded":"2024-09-16T04:09:08.457364+0000","last_fullsized":"2024-09-16T04:09:08.457364+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-09-16T04:08:23.448368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.448368+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:53.308793+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0.011966566,"stat_sum":{"num_bytes":196608,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":1154,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6,3,7,8,0,5],"acting":[11,9,6,3,7,8,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.1","version":"0'0","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.540795+0000","last_change":"2024-09-16T04:07:53.018517+0000","last_active":"2024-09-16T04:09:09.540795+0000","last_peered":"2024-09-16T04:09:09.540795+0000","last_clean":"2024-09-16T04:09:09.540795+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:09:09.540795+0000","last_undegraded":"2024-09-16T04:09:09.540795+0000","last_fullsized":"2024-09-16T04:09:09.540795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:07:49.925832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:17.259564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'34","reported_seq":374,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.448463+0000","last_change":"2024-09-16T04:09:06.449833+0000","last_active":"2024-09-16T04:09:06.448463+0000","last_peered":"2024-09-16T04:09:06.448463+0000","last_clean":"2024-09-16T04:09:06.448463+0000","last_became_active":"2024-09-16T04:08:32.459260+0000","last_became_peered":"2024-09-16T04:08:32.459260+0000","last_unstale":"2024-09-16T04:09:06.448463+0000","last_undegraded":"2024-09-16T04:09:06.448463+0000","last_fullsized":"2024-09-16T04:09:06.448463+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'15","last_scrub_stamp":"2024-09-16T04:08:22.680404+0000","last_deep_scrub":"45'15","last_deep_scrub_stamp":"2024-09-16T04:08:22.680404+0000","last_clean_scrub_stamp":"2024-09-16T04:08:22.680404+0000","objects_scrubbed":1,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:41.929432+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.002048542,"stat_sum":{"num_bytes":1785856,"num_objects":3,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":28,"num_write_kb":3760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10,11,5,9,6,7],"acting":[0,4,10,11,5,9,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.450229+0000","last_change":"2024-09-16T04:08:58.876711+0000","last_active":"2024-09-16T04:09:08.450229+0000","last_peered":"2024-09-16T04:09:08.450229+0000","last_clean":"2024-09-16T04:09:08.450229+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:09:08.450229+0000","last_undegraded":"2024-09-16T04:09:08.450229+0000","last_fullsized":"2024-09-16T04:09:08.450229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_clean_scrub_stamp":"2024-09-16T04:08:58.876583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:28.287717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'119","reported_seq":511,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.859542+0000","last_change":"2024-09-16T04:09:08.449008+0000","last_active":"2024-09-16T04:09:09.859542+0000","last_peered":"2024-09-16T04:09:09.859542+0000","last_clean":"2024-09-16T04:09:09.859542+0000","last_became_active":"2024-09-16T04:08:32.466786+0000","last_became_peered":"2024-09-16T04:08:32.466786+0000","last_unstale":"2024-09-16T04:09:09.859542+0000","last_undegraded":"2024-09-16T04:09:09.859542+0000","last_fullsized":"2024-09-16T04:09:09.859542+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'52","last_scrub_stamp":"2024-09-16T04:08:23.356294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.356294+0000","objects_scrubbed":3,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:39.907457+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0.0023891390000000002,"stat_sum":{"num_bytes":9109504,"num_objects":5,"num_object_clones":2,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":25162,"num_write":62,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":6144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6,4,5,7,8,10],"acting":[3,9,6,4,5,7,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"2.3","version":"21'1","reported_seq":170,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444559+0000","last_change":"2024-09-16T04:08:59.147922+0000","last_active":"2024-09-16T04:09:09.444559+0000","last_peered":"2024-09-16T04:09:09.444559+0000","last_clean":"2024-09-16T04:09:09.444559+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:09:09.444559+0000","last_undegraded":"2024-09-16T04:09:09.444559+0000","last_fullsized":"2024-09-16T04:09:09.444559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-09-16T04:08:59.147797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:59.147797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:25.871032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":335,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.455690+0000","last_change":"2024-09-16T04:08:55.861559+0000","last_active":"2024-09-16T04:09:08.455690+0000","last_peered":"2024-09-16T04:09:08.455690+0000","last_clean":"2024-09-16T04:09:08.455690+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:09:08.455690+0000","last_undegraded":"2024-09-16T04:09:08.455690+0000","last_fullsized":"2024-09-16T04:09:08.455690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'137","last_scrub_stamp":"2024-09-16T04:08:55.861468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:08:55.861468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:57.766582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"92'66","reported_seq":441,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.446601+0000","last_change":"2024-09-16T04:09:08.447541+0000","last_active":"2024-09-16T04:09:08.446601+0000","last_peered":"2024-09-16T04:09:08.446601+0000","last_clean":"2024-09-16T04:09:08.446601+0000","last_became_active":"2024-09-16T04:08:32.466625+0000","last_became_peered":"2024-09-16T04:08:32.466625+0000","last_unstale":"2024-09-16T04:09:08.446601+0000","last_undegraded":"2024-09-16T04:09:08.446601+0000","last_fullsized":"2024-09-16T04:09:08.446601+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'38","last_scrub_stamp":"2024-09-16T04:08:24.203960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:24.203960+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:44.293708+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0.0012348089999999999,"stat_sum":{"num_bytes":5554176,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":26212,"num_write":10,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5390336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2,4,3,5,10,8],"acting":[11,7,2,4,3,5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.9","version":"93'119","reported_seq":489,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.459597+0000","last_change":"2024-09-16T04:09:08.462883+0000","last_active":"2024-09-16T04:09:08.459597+0000","last_peered":"2024-09-16T04:09:08.459597+0000","last_clean":"2024-09-16T04:09:08.459597+0000","last_became_active":"2024-09-16T04:08:32.454824+0000","last_became_peered":"2024-09-16T04:08:32.454824+0000","last_unstale":"2024-09-16T04:09:08.459597+0000","last_undegraded":"2024-09-16T04:09:08.459597+0000","last_fullsized":"2024-09-16T04:09:08.459597+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'39","last_scrub_stamp":"2024-09-16T04:08:23.214261+0000","last_deep_scrub":"45'39","last_deep_scrub_stamp":"2024-09-16T04:08:23.214261+0000","last_clean_scrub_stamp":"2024-09-16T04:08:23.214261+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:32.700919+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0.017477276,"stat_sum":{"num_bytes":12943360,"num_objects":9,"num_object_clones":6,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":14,"num_read_kb":21317,"num_write":24,"num_write_kb":6304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11059200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7,9,5,6,4,10],"acting":[8,11,7,9,5,6,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.a","version":"93'69","reported_seq":418,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.443753+0000","last_change":"2024-09-16T04:09:08.494544+0000","last_active":"2024-09-16T04:09:09.443753+0000","last_peered":"2024-09-16T04:09:09.443753+0000","last_clean":"2024-09-16T04:09:09.443753+0000","last_became_active":"2024-09-16T04:08:32.458689+0000","last_became_peered":"2024-09-16T04:08:32.458689+0000","last_unstale":"2024-09-16T04:09:09.443753+0000","last_undegraded":"2024-09-16T04:09:09.443753+0000","last_fullsized":"2024-09-16T04:09:09.443753+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:28.335645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.048377525999999997,"stat_sum":{"num_bytes":9338880,"num_objects":6,"num_object_clones":4,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":16,"num_read_kb":5,"num_write":24,"num_write_kb":13584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4620288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5,10,8,9,3,7],"acting":[6,4,5,10,8,9,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.b","version":"93'102","reported_seq":487,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.859705+0000","last_change":"2024-09-16T04:09:08.450149+0000","last_active":"2024-09-16T04:09:09.859705+0000","last_peered":"2024-09-16T04:09:09.859705+0000","last_clean":"2024-09-16T04:09:09.859705+0000","last_became_active":"2024-09-16T04:08:32.458688+0000","last_became_peered":"2024-09-16T04:08:32.458688+0000","last_unstale":"2024-09-16T04:09:09.859705+0000","last_undegraded":"2024-09-16T04:09:09.859705+0000","last_fullsized":"2024-09-16T04:09:09.859705+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:31.699655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027560810000000001,"stat_sum":{"num_bytes":12632064,"num_objects":8,"num_object_clones":4,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":56481,"num_write":72,"num_write_kb":13632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3620864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4,8,11,7,6,9],"acting":[3,0,4,8,11,7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.c","version":"93'95","reported_seq":445,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.444870+0000","last_change":"2024-09-16T04:09:08.478728+0000","last_active":"2024-09-16T04:09:09.444870+0000","last_peered":"2024-09-16T04:09:09.444870+0000","last_clean":"2024-09-16T04:09:09.444870+0000","last_became_active":"2024-09-16T04:08:03.222541+0000","last_became_peered":"2024-09-16T04:08:03.222541+0000","last_unstale":"2024-09-16T04:09:09.444870+0000","last_undegraded":"2024-09-16T04:09:09.444870+0000","last_fullsized":"2024-09-16T04:09:09.444870+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:31.031193+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.033256309999999997,"stat_sum":{"num_bytes":6832128,"num_objects":9,"num_object_clones":4,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":46,"num_read_kb":7125,"num_write":97,"num_write_kb":10336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6,10,9,2,0,11],"acting":[5,3,6,10,9,2,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.d","version":"92'46","reported_seq":434,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.457832+0000","last_change":"2024-09-16T04:09:08.458188+0000","last_active":"2024-09-16T04:09:08.457832+0000","last_peered":"2024-09-16T04:09:08.457832+0000","last_clean":"2024-09-16T04:09:08.457832+0000","last_became_active":"2024-09-16T04:08:32.461735+0000","last_became_peered":"2024-09-16T04:08:32.461735+0000","last_unstale":"2024-09-16T04:09:08.457832+0000","last_undegraded":"2024-09-16T04:09:08.457832+0000","last_fullsized":"2024-09-16T04:09:08.457832+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:24.263946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073052399999999997,"stat_sum":{"num_bytes":1818624,"num_objects":5,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":243,"num_write":40,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2473984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9,6,11,3,4,0],"acting":[7,5,9,6,11,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.e","version":"69'19","reported_seq":360,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.446189+0000","last_change":"2024-09-16T04:09:08.446541+0000","last_active":"2024-09-16T04:09:08.446189+0000","last_peered":"2024-09-16T04:09:08.446189+0000","last_clean":"2024-09-16T04:09:08.446189+0000","last_became_active":"2024-09-16T04:08:32.457538+0000","last_became_peered":"2024-09-16T04:08:32.457538+0000","last_unstale":"2024-09-16T04:09:08.446189+0000","last_undegraded":"2024-09-16T04:09:08.446189+0000","last_fullsized":"2024-09-16T04:09:08.446189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:02.050701+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:26.691248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053797600000000001,"stat_sum":{"num_bytes":3817472,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":22372,"num_write":7,"num_write_kb":4208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6,4,0,9,3,10],"acting":[11,8,6,4,0,9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.f","version":"92'18","reported_seq":348,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.450241+0000","last_change":"2024-09-16T04:09:08.451194+0000","last_active":"2024-09-16T04:09:08.450241+0000","last_peered":"2024-09-16T04:09:08.450241+0000","last_clean":"2024-09-16T04:09:08.450241+0000","last_became_active":"2024-09-16T04:08:03.221547+0000","last_became_peered":"2024-09-16T04:08:03.221547+0000","last_unstale":"2024-09-16T04:09:08.450241+0000","last_undegraded":"2024-09-16T04:09:08.450241+0000","last_fullsized":"2024-09-16T04:09:08.450241+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'16","last_scrub_stamp":"2024-09-16T04:09:06.936150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:09:06.936150+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:17.559188+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.001125734,"stat_sum":{"num_bytes":1474560,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8,2,9,10,0,4],"acting":[7,11,8,2,9,10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.10","version":"30'2","reported_seq":254,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T04:09:09.443229+0000","last_change":"2024-09-16T04:09:08.447028+0000","last_active":"2024-09-16T04:09:09.443229+0000","last_peered":"2024-09-16T04:09:09.443229+0000","last_clean":"2024-09-16T04:09:09.443229+0000","last_became_active":"2024-09-16T04:08:32.459694+0000","last_became_peered":"2024-09-16T04:08:32.459694+0000","last_unstale":"2024-09-16T04:09:09.443229+0000","last_undegraded":"2024-09-16T04:09:09.443229+0000","last_fullsized":"2024-09-16T04:09:09.443229+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"30'2","last_scrub_stamp":"2024-09-16T04:08:34.079799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.079799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:46.373607+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0012269049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5,3,0,2,10,7],"acting":[6,9,5,3,0,2,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.11","version":"80'32","reported_seq":283,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.454746+0000","last_change":"2024-09-16T04:09:08.456883+0000","last_active":"2024-09-16T04:09:08.454746+0000","last_peered":"2024-09-16T04:09:08.454746+0000","last_clean":"2024-09-16T04:09:08.454746+0000","last_became_active":"2024-09-16T04:08:32.455596+0000","last_became_peered":"2024-09-16T04:08:32.455596+0000","last_unstale":"2024-09-16T04:09:08.454746+0000","last_undegraded":"2024-09-16T04:09:08.454746+0000","last_fullsized":"2024-09-16T04:09:08.454746+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'15","last_scrub_stamp":"2024-09-16T04:08:34.403063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.403063+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:40.573546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0023664620000000002,"stat_sum":{"num_bytes":2146304,"num_objects":1,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6364,"num_write":26,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6,0,3,9,8,11],"acting":[10,4,6,0,3,9,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]},{"pgid":"3.12","version":"89'119","reported_seq":397,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T04:09:06.450622+0000","last_change":"2024-09-16T04:09:06.451427+0000","last_active":"2024-09-16T04:09:06.450622+0000","last_peered":"2024-09-16T04:09:06.450622+0000","last_clean":"2024-09-16T04:09:06.450622+0000","last_became_active":"2024-09-16T04:08:32.460980+0000","last_became_peered":"2024-09-16T04:08:32.460980+0000","last_unstale":"2024-09-16T04:09:06.450622+0000","last_undegraded":"2024-09-16T04:09:06.450622+0000","last_fullsized":"2024-09-16T04:09:06.450622+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'62","last_scrub_stamp":"2024-09-16T04:08:35.178702+0000","last_deep_scrub":"57'62","last_deep_scrub_stamp":"2024-09-16T04:08:35.178702+0000","last_clean_scrub_stamp":"2024-09-16T04:08:35.178702+0000","objects_scrubbed":6,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:52.750345+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0.002257359,"stat_sum":{"num_bytes":9420800,"num_objects":8,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":10,"num_read_kb":1857,"num_write":16,"num_write_kb":5024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,8,2,5,6,11],"acting":[0,7,3,8,2,5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"2"}]},{"pgid":"3.13","version":"93'36","reported_seq":286,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T04:09:08.460931+0000","last_change":"2024-09-16T04:09:08.462370+0000","last_active":"2024-09-16T04:09:08.460931+0000","last_peered":"2024-09-16T04:09:08.460931+0000","last_clean":"2024-09-16T04:09:08.460931+0000","last_became_active":"2024-09-16T04:08:32.463080+0000","last_became_peered":"2024-09-16T04:08:32.463080+0000","last_unstale":"2024-09-16T04:09:08.460931+0000","last_undegraded":"2024-09-16T04:09:08.460931+0000","last_fullsized":"2024-09-16T04:09:08.460931+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"37'16","last_scrub_stamp":"2024-09-16T04:08:34.081657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:02.050701+0000","last_clean_scrub_stamp":"2024-09-16T04:08:34.081657+0000","objects_scrubbed":1,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:01.908123+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.012386222000000001,"stat_sum":{"num_bytes":1851392,"num_objects":2,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4,2,6,3,9,8],"acting":[7,5,4,2,6,3,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"},{"start":"1b","length":"1"},{"start":"1e","length":"4"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":132546560,"num_objects":100,"num_object_clones":53,"num_object_copies":800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":11,"num_read":255,"num_read_kb":202225,"num_write":610,"num_write_kb":119744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":45367296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270499840,"data_stored":377106432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":585,"internal_metadata":0},"log_size":1514,"ondisk_log_size":1514,"up":208,"acting":208,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378643,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77600,"kb_used_data":28364,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94294240,"statfs":{"total":96636764160,"available":96557301760,"internally_reserved":0,"allocated":29044736,"data_stored":38233732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378642,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79312,"kb_used_data":25908,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94292528,"statfs":{"total":96636764160,"available":96555548672,"internally_reserved":0,"allocated":26529792,"data_stored":36288132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378642,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75668,"kb_used_data":26424,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94296172,"statfs":{"total":96636764160,"available":96559280128,"internally_reserved":0,"allocated":27058176,"data_stored":36292209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378645,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77204,"kb_used_data":23800,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94294636,"statfs":{"total":96636764160,"available":96557707264,"internally_reserved":0,"allocated":24371200,"data_stored":31786609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51628,"kb_used_data":16280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320212,"statfs":{"total":96636764160,"available":96583897088,"internally_reserved":0,"allocated":16670720,"data_stored":22951537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411346,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79932,"kb_used_data":26528,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94291908,"statfs":{"total":96636764160,"available":96554913792,"internally_reserved":0,"allocated":27164672,"data_stored":42420035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411348,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52488,"kb_used_data":21296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319352,"statfs":{"total":96636764160,"available":96583016448,"internally_reserved":0,"allocated":21807104,"data_stored":29578865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411346,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72992,"kb_used_data":23752,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94298848,"statfs":{"total":96636764160,"available":96562020352,"internally_reserved":0,"allocated":24322048,"data_stored":34497827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411346,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55620,"kb_used_data":24428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316220,"statfs":{"total":96636764160,"available":96579809280,"internally_reserved":0,"allocated":25014272,"data_stored":33543793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411348,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58896,"kb_used_data":27700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94312944,"statfs":{"total":96636764160,"available":96576454656,"internally_reserved":0,"allocated":28364800,"data_stored":42628931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":0,"seq":326417514500,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444051,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55480,"kb_used_data":20128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316360,"statfs":{"total":96636764160,"available":96579952640,"internally_reserved":0,"allocated":20611072,"data_stored":31904709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19554304,"data_stored":31244288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10178560,"data_stored":12312576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":585,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15589376,"data_stored":22274048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23289856,"data_stored":31109120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25976832,"data_stored":35614720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25444352,"data_stored":35610624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27959296,"data_stored":37556224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26763264,"data_stored":38789120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":23932928,"data_stored":32866304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24387584,"data_stored":32493568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":20725760,"data_stored":28901376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26697728,"data_stored":38334464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:09:11.387 INFO:tasks.thrashosds.thrasher:Setting 3.11 to [9, 3, 4, 0, 7, 2, 5, 11, 10, 1, 6, 8] 2024-09-16T04:09:11.387 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.11', '9', '3', '4', '0', '7', '2', '5', '11', '10', '1', '6', '8'] 2024-09-16T04:09:11.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.11 9 3 4 0 7 2 5 11 10 1 6 8 2024-09-16T04:09:11.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T04:09:11.431 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:11.428+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:11.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T04:09:11.478 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2024-09-16T04:09:11.479 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T04:09:11.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T04:09:11.533 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-16T04:09:11.531+0000 7f306f8cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33077) UID: 0 2024-09-16T04:09:11.575 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2024-09-16T04:09:11.576 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T04:09:11.599 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2024-09-16T04:09:11.600 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T04:09:11.622 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2024-09-16T04:09:11.622 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T04:09:11.632 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:11.631+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:11.643 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2024-09-16T04:09:11.644 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDECSubOpWrite(3.3s1 32/27 ECSubWrite(tid=11, reqid=client.4440.0:199, at_version=32'8, trim_to=0'0, roll_forward_to=30'6))", 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:08.224890+0000", 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.431720769000002, 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.46824734299999998, 2024-09-16T04:09:11.661 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.224890+0000", 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.224890+0000", 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.224891+0000", 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6199999999999996e-07 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.225386+0000", 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049458200000000003 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.691919+0000", 2024-09-16T04:09:11.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.466533643 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.691926+0000", 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6139999999999997e-06 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.691978+0000", 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1987000000000002e-05 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_started", 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.691985+0000", 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4919999999999998e-06 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_committed", 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.693106+0000", 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011215649999999999 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:08.693138+0000", 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1597999999999998e-05 2024-09-16T04:09:11.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:202 3.7s0 3:eb2a7864:::smithi01639990-11:head [copy-from ver 6] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:09.090975+0000", 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 62.565635604999997, 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032721588000000003, 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 202 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.090975+0000", 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.090975+0000", 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.090979+0000", 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.090992+0000", 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3229e-05 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091027+0000", 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4715999999999998e-05 2024-09-16T04:09:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091043+0000", 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5792000000000001e-05 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091155+0000", 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112506 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-16T04:09:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091165+0000", 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1519999999999998e-06 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.093362+0000", 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021971989999999999 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.093474+0000", 2024-09-16T04:09:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111964 2024-09-16T04:09:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_started", 2024-09-16T04:09:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.113681+0000", 2024-09-16T04:09:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020207253000000001 2024-09-16T04:09:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_committed", 2024-09-16T04:09:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.115749+0000", 2024-09-16T04:09:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002068351 2024-09-16T04:09:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.123651+0000", 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079019380000000007 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.123697+0000", 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5836e-05 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:203 3.7s0 3:eb2a7864:::smithi01639990-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:09:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:09.091081+0000", 2024-09-16T04:09:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 62.565529882, 2024-09-16T04:09:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023789853999999999, 2024-09-16T04:09:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-16T04:09:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 203 2024-09-16T04:09:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091081+0000", 2024-09-16T04:09:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091081+0000", 2024-09-16T04:09:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091082+0000", 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6700000000000003e-07 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091087+0000", 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3190000000000002e-06 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091100+0000", 2024-09-16T04:09:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2860999999999999e-05 2024-09-16T04:09:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091107+0000", 2024-09-16T04:09:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3679999999999996e-06 2024-09-16T04:09:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091305+0000", 2024-09-16T04:09:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019742800000000001 2024-09-16T04:09:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.091312+0000", 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9990000000000002e-06 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.093749+0000", 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002437491 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.093803+0000", 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4027000000000003e-05 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.114630+0000", 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02082637 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.114743+0000", 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011294300000000001 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.114871+0000", 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000128381 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDECSubOpWrite(3.as6 33/27 ECSubWrite(tid=14, reqid=osd.6.0:14, at_version=33'10, trim_to=0'0, roll_forward_to=32'8))", 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:09.094996+0000", 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 62.561614853000002, 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.048682581000000003, 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.094996+0000", 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.094996+0000", 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.094996+0000", 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5300000000000001e-07 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.095031+0000", 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4421000000000001e-05 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.143019+0000", 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047988537999999997 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.143026+0000", 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2890000000000003e-06 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.143057+0000", 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1520000000000003e-05 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_started", 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.143062+0000", 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.561e-06 2024-09-16T04:09:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_committed", 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.143654+0000", 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059178900000000001 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:09.143679+0000", 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.531e-05 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:356 3.2s0 3:4e1a0922:::smithi01639990-46:head [setxattr _header (56) in=7b] snapc 8=[8,6] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:17.588028+0000", 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.068583259999997, 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.84102941099999995, 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 356 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588028+0000", 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588028+0000", 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588029+0000", 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0900000000000001e-07 2024-09-16T04:09:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588032+0000", 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8240000000000001e-06 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588039+0000", 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0149999999999998e-06 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588043+0000", 2024-09-16T04:09:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7390000000000001e-06 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.589275+0000", 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001232138 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.589444+0000", 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016890500000000001 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_started", 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.590034+0000", 2024-09-16T04:09:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00058961500000000004 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_committed", 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.590830+0000", 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079578799999999996 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:18.428943+0000", 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.83811363400000005 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:18.429057+0000", 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011404399999999999 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:357 3.2s0 3:4e1a0922:::smithi01639990-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:17.588053+0000", 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.068558271000001, 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3004059800000001, 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-16T04:09:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 357 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588053+0000", 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588053+0000", 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588053+0000", 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7999999999999996e-07 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588056+0000", 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.723e-06 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588063+0000", 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3579999999999997e-06 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.588069+0000", 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9290000000000003e-06 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.590844+0000", 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027749580000000001 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:17.590898+0000", 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3672999999999999e-05 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:18.429555+0000", 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.83865689499999996 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:18.429616+0000", 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0999000000000003e-05 2024-09-16T04:09:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:18.888459+0000", 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.45884296499999999 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:434 3.7s0 3:e711f10d:::smithi01639990-28:head [delete] snapc d=[d,c,b] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:27.508427+0000", 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.148184270999998, 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.84610548699999999, 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 434 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.508427+0000", 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.508427+0000", 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.508427+0000", 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.72e-07 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.508429+0000", 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2110000000000001e-06 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.508435+0000", 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8799999999999996e-06 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.508439+0000", 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3299999999999999e-06 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.508491+0000", 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2349000000000003e-05 2024-09-16T04:09:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.508592+0000", 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010132 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_started", 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.509441+0000", 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00084823699999999995 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_committed", 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:27.510837+0000", 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001396174 2024-09-16T04:09:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.354469+0000", 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.84363268499999999 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.354532+0000", 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2928999999999993e-05 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:452 3.2s0 3:47876147:::smithi01639990-37:head [setxattr _header (57) in=7b] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:28.516619+0000", 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.139991868999999, 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.52209116200000005, 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 452 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516619+0000", 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516619+0000", 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516620+0000", 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8599999999999997e-07 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516628+0000", 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7060000000000004e-06 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516636+0000", 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3969999999999994e-06 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516642+0000", 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8699999999999997e-06 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.518782+0000", 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00213999 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.518946+0000", 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016358900000000001 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_started", 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.519495+0000", 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00054913100000000001 2024-09-16T04:09:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_committed", 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.520624+0000", 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011297600000000001 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:29.038617+0000", 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.517992601 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:29.038710+0000", 2024-09-16T04:09:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3331999999999993e-05 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:453 3.2s0 3:47876147:::smithi01639990-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:28.516655+0000", 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.139955880000002, 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.52351270100000002, 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 453 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516655+0000", 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516655+0000", 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516656+0000", 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9400000000000001e-07 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516658+0000", 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4159999999999998e-06 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516665+0000", 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6440000000000003e-06 2024-09-16T04:09:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.516670+0000", 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0459999999999996e-06 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.520172+0000", 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003502607 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:28.520235+0000", 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2632999999999995e-05 2024-09-16T04:09:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:29.039070+0000", 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.51883495000000002 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:29.039142+0000", 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2082999999999994e-05 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:29.040168+0000", 2024-09-16T04:09:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001025928 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(3.2s0 57/56 [PushReplyOp(3:47876147:::smithi01639990-37:head)])", 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:32.559967+0000", 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.096643886999999, 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040171806999999997, 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.559967+0000", 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.559967+0000", 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.559968+0000", 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7700000000000004e-07 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.559972+0000", 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3599999999999998e-06 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.599259+0000", 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039287080000000002 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.599268+0000", 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4230000000000005e-06 2024-09-16T04:09:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.599338+0000", 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0600999999999997e-05 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.600139+0000", 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080076599999999996 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(3.2s0 57/56 [PushReplyOp(3:4376f790:::smithi01639990-22:head)])", 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:32.619846+0000", 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.036765428000002, 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.28549633899999999, 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.619846+0000", 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.619846+0000", 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.619846+0000", 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5999999999999999e-07 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.619850+0000", 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.563e-06 2024-09-16T04:09:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.904738+0000", 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.28488804899999998 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.904758+0000", 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0755999999999999e-05 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.904921+0000", 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016257399999999999 2024-09-16T04:09:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:32.905342+0000", 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042093700000000003 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDScrubReserve(3.17s3,QREQUEST,e:57,from:0(0),reservation_nonce:3)", 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:33.713968+0000", 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.942642630999998, 2024-09-16T04:09:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.089086057999999996, 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:33.713968+0000", 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:33.713968+0000", 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:33.713969+0000", 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5600000000000005e-07 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:33.713973+0000", 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0759999999999996e-06 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:33.802916+0000", 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.088942503000000006 2024-09-16T04:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:33.802921+0000", 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9559999999999996e-06 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:33.803011+0000", 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0594999999999996e-05 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:33.803028+0000", 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6787999999999999e-05 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:33.803055+0000", 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6384e-05 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:514 3.bs0 3:dbca5599:::smithi01639990-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:37.587830+0000", 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.068781022000003, 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043606001999999998, 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 514 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.587830+0000", 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.587830+0000", 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.587830+0000", 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4600000000000001e-07 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.587832+0000", 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.336e-06 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.587836+0000", 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8500000000000004e-06 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.587838+0000", 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7089999999999998e-06 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.591260+0000", 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034213410000000001 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.591309+0000", 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9725999999999998e-05 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.596477+0000", 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051679930000000001 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.596565+0000", 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7682999999999994e-05 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.631436+0000", 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034871118 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:529 3.7s0 3:e711f10d:::smithi01639990-28:head [delete] snapc 12=[12,11,10] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:37.630572+0000", 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 34.026038690999997, 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.145044438, 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 529 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.630572+0000", 2024-09-16T04:09:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.630572+0000", 2024-09-16T04:09:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.630574+0000", 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0200000000000001e-06 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.630583+0000", 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0210000000000005e-06 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.630601+0000", 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7107000000000001e-05 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.630609+0000", 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.630670+0000", 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0538999999999998e-05 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.630753+0000", 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2911999999999995e-05 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:37.775617+0000", 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.144864249 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:535 3.7s0 3:e76262c7:::smithi01639990-15:head [copy-from ver 35] snapc 13=[13,12,10] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:39.514992+0000", 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.141618794000003, 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026410659999999999, 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 535 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.514992+0000", 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.514992+0000", 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.514995+0000", 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.373e-06 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515005+0000", 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0427e-05 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515029+0000", 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3787000000000001e-05 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515043+0000", 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3702e-05 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515142+0000", 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9345999999999994e-05 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op must wait for map", 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515150+0000", 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.259e-06 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.518254+0000", 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031033250000000001 2024-09-16T04:09:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.518348+0000", 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4628000000000003e-05 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_started", 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.535030+0000", 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016681818000000001 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_committed", 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.538250+0000", 2024-09-16T04:09:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003220086 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.541347+0000", 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030971810000000001 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.541403+0000", 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5727999999999999e-05 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4440.0:536 3.7s0 3:e76262c7:::smithi01639990-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:39.515088+0000", 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.141522848999998, 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022913532, 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4440", 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 536 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515088+0000", 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515088+0000", 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515089+0000", 2024-09-16T04:09:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1699999999999997e-07 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515095+0000", 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5430000000000001e-06 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515104+0000", 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.302e-06 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515111+0000", 2024-09-16T04:09:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.345e-06 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515225+0000", 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011406800000000001 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.515233+0000", 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5000000000000002e-06 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.518537+0000", 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033040449999999998 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.518595+0000", 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8671000000000001e-05 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.537764+0000", 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01916839 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.537858+0000", 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4030999999999995e-05 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:39.538002+0000", 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014391999999999999 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 66/56 ECSubWriteReply(tid=317, last_complete=66'90, committed=1, applied=1))", 2024-09-16T04:09:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:41.582232+0000", 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.074379527000001, 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.72521073800000002, 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:41.582232+0000", 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:41.582232+0000", 2024-09-16T04:09:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:41.582232+0000", 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:41.582235+0000", 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.005e-06 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:42.306932+0000", 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.72469726999999995 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:42.306947+0000", 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.43e-05 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:42.307007+0000", 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0541000000000002e-05 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:42.307442+0000", 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00043506300000000002 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 68/56 ECSubWriteReply(tid=335, last_complete=68'96, committed=1, applied=1))", 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:43.564798+0000", 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.091812779000001, 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33829203000000002, 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:43.564798+0000", 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:43.564798+0000", 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:43.564799+0000", 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8099999999999999e-07 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:43.564801+0000", 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3750000000000001e-06 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:43.902953+0000", 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33815231499999998 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:43.902965+0000", 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1865000000000001e-05 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:43.903059+0000", 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3636999999999998e-05 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:43.903090+0000", 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1557000000000003e-05 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDECSubOpWrite(3.12s2 77/56 ECSubWrite(tid=178, reqid=client.4440.0:742, at_version=77'96, trim_to=0'0, roll_forward_to=77'93))", 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:52.468588+0000", 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.188023136000002, 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023695197000000001, 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.468588+0000", 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.468588+0000", 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.468589+0000", 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.06e-07 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.468607+0000", 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8287e-05 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.491767+0000", 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023160511000000002 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.491773+0000", 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9340000000000003e-06 2024-09-16T04:09:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.491814+0000", 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0224e-05 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_started", 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.491820+0000", 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9850000000000003e-06 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_committed", 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.492250+0000", 2024-09-16T04:09:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00043022399999999999 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:52.492283+0000", 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3325999999999999e-05 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 81/56 ECSubWriteReply(tid=412, last_complete=81'113, committed=1, applied=1))", 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2024-09-16T04:08:56.500882+0000", 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 15.155728735, 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.20251751600000001, 2024-09-16T04:09:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "continuous": false, 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:56.500882+0000", 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:56.500882+0000", 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:56.500883+0000", 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0600000000000001e-07 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:56.500885+0000", 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5110000000000002e-06 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:56.702931+0000", 2024-09-16T04:09:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.202045539 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:56.702943+0000", 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2544999999999999e-05 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:56.703010+0000", 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6538000000000006e-05 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2024-09-16T04:08:56.703400+0000", 2024-09-16T04:09:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039007700000000001 2024-09-16T04:09:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-09-16T04:09:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-09-16T04:09:11.762 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:11.763 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2024-09-16T04:09:11.763 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T04:09:11.764 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T04:09:11.764 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2024-09-16T04:09:11.765 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T04:09:11.765 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6: {} 2024-09-16T04:09:11.765 INFO:teuthology.orchestra.run.smithi016.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T04:09:11.766 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7: {} 2024-09-16T04:09:11.766 INFO:teuthology.orchestra.run.smithi016.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T04:09:11.766 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8: {} 2024-09-16T04:09:11.766 INFO:teuthology.orchestra.run.smithi016.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T04:09:11.767 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-09-16T04:09:11.762+0000 7f82d5cfc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33130) UID: 0 2024-09-16T04:09:11.777 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9: {} 2024-09-16T04:09:11.777 INFO:teuthology.orchestra.run.smithi016.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T04:09:11.798 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10: {} 2024-09-16T04:09:11.798 INFO:teuthology.orchestra.run.smithi016.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T04:09:11.820 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11: {} 2024-09-16T04:09:11.821 INFO:teuthology.orchestra.run.smithi016.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T04:09:11.864 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-09-16T04:09:11.862+0000 7fe1cea84640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33133) UID: 0 2024-09-16T04:09:11.882 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:11.883 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:11.883 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:09:11.883 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:11.892 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T04:09:11.965 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:11.963+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:12.036 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:12.036 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:12.037 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:12.037 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:12.037 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:12.045 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T04:09:12.066 INFO:tasks.ceph.osd.8.smithi194.stderr:2024-09-16T04:09:12.064+0000 7f9d70ad2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33290) UID: 0 2024-09-16T04:09:12.151 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:12.151 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:12.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:12.151 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:12.151 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.151 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:199 3.3s0 3:c05d4807:::smithi01639990-31:head [copy-from ver 11] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:09:12.151 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:09:12.151 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 63.938307709999997, 2024-09-16T04:09:12.151 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.48371305599999997, 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 199 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209956+0000", 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209959+0000", 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7460000000000001e-06 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.153 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.209971+0000", 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2787e-05 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210000+0000", 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8952999999999999e-05 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.154 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210015+0000", 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.434e-05 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210169+0000", 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015455199999999999 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210317+0000", 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000147291 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.155 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.224854+0000", 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014536926 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226325+0000", 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001471099 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693612+0000", 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.467287324 2024-09-16T04:09:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.693669+0000", 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7037999999999998e-05 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:200 3.3s0 3:c05d4807:::smithi01639990-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 63.938217420999997, 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016139113, 2024-09-16T04:09:12.157 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 200 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.158 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210047+0000", 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210051+0000", 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8800000000000001e-06 2024-09-16T04:09:12.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210065+0000", 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3567000000000001e-05 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210074+0000", 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2399999999999996e-06 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210464+0000", 2024-09-16T04:09:12.160 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00038978800000000002 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.210496+0000", 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2539999999999997e-05 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.225987+0000", 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015491398999999999 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.161 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226084+0000", 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6618000000000002e-05 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.226185+0000", 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000101205 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s6 33/27 ECSubWrite(tid=38, reqid=osd.8.0:38, at_version=33'27, trim_to=0'0, roll_forward_to=32'23))", 2024-09-16T04:09:12.162 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 63.048673481999998, 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010829642, 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.163 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099590+0000", 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099591+0000", 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4399999999999999e-07 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099619+0000", 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7555e-05 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.164 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099655+0000", 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6337000000000001e-05 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099691+0000", 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5994000000000001e-05 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099819+0000", 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000128484 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.165 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.099829+0000", 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1949999999999992e-06 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110366+0000", 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010537504 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.110420+0000", 2024-09-16T04:09:12.166 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3729e-05 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s2 33/27 ECSubWrite(tid=16, reqid=osd.0.0:16, at_version=33'8, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 63.046856849999998, 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012298224, 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.167 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101407+0000", 2024-09-16T04:09:12.168 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0900000000000002e-07 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101422+0000", 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5004000000000001e-05 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101446+0000", 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3632999999999999e-05 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.169 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101452+0000", 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0260000000000004e-06 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101511+0000", 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9515000000000002e-05 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101523+0000", 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1094000000000001e-05 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.170 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113680+0000", 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012157401 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113705+0000", 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5041999999999998e-05 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.es3 33/27 ECSubWrite(tid=31, reqid=osd.11.0:31, at_version=33'6, trim_to=0'0, roll_forward_to=32'4))", 2024-09-16T04:09:12.171 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 63.046738499999996, 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012148897000000001, 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.172 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101525+0000", 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101526+0000", 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0900000000000001e-07 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101543+0000", 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7215999999999999e-05 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.173 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101584+0000", 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1352e-05 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101590+0000", 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5779999999999998e-06 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101650+0000", 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0093000000000002e-05 2024-09-16T04:09:12.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101662+0000", 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2257e-05 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113631+0000", 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.01196856 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.175 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113674+0000", 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3132e-05 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=30, reqid=osd.11.0:30, at_version=33'16, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 63.046715484000003, 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012128856, 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.176 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.177 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101548+0000", 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2399999999999999e-07 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101558+0000", 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1980000000000007e-06 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101618+0000", 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0911e-05 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.178 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101624+0000", 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6899999999999997e-06 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101690+0000", 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5988000000000006e-05 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.101701+0000", 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0878e-05 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113645+0000", 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011944110000000001 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.113677+0000", 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1856999999999997e-05 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.8s3 33/27 ECSubWrite(tid=32, reqid=osd.11.0:32, at_version=33'18, trim_to=0'0, roll_forward_to=32'14))", 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 63.045236430000003, 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014214867000000001, 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103027+0000", 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103028+0000", 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8999999999999995e-07 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103051+0000", 2024-09-16T04:09:12.182 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2909999999999999e-05 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103122+0000", 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1339000000000003e-05 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103130+0000", 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8790000000000003e-06 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.183 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104942+0000", 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001811893 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104952+0000", 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5449999999999992e-06 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117201+0000", 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012249395 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.184 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117242+0000", 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1016000000000002e-05 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s6 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 63.043708870000003, 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012207213999999999, 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.185 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104555+0000", 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.186 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104556+0000", 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8999999999999995e-07 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104963+0000", 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000407604 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105023+0000", 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9684e-05 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.187 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105031+0000", 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3599999999999996e-06 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105239+0000", 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00020760900000000001 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105251+0000", 2024-09-16T04:09:12.188 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1788e-05 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116731+0000", 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011480292 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116762+0000", 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1087000000000002e-05 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 51/48 ECSubWriteReply(tid=36, last_complete=51'22, committed=1, applied=1))", 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:26.486527+0000", 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.661736617999999, 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.69968956299999996, 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486527+0000", 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.190 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486527+0000", 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486528+0000", 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5899999999999995e-07 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:26.486532+0000", 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4449999999999998e-06 2024-09-16T04:09:12.191 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.185640+0000", 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.69910766300000005 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.185651+0000", 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1423999999999999e-05 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.185718+0000", 2024-09-16T04:09:12.192 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6724999999999996e-05 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.186217+0000", 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049854700000000001 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpRead(3.as1 57/56 ECSubRead(tid=39, to_read={3:5e52341f:::smithi01639990-38:d=0,262144,0}, subchunks={3:5e52341f:::smithi01639990-38:d=[0,1]}, attrs_to_read=))", 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.575872+0000", 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 39.572391637999999, 2024-09-16T04:09:12.193 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.86215607299999997, 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.575872+0000", 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.575872+0000", 2024-09-16T04:09:12.194 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.575873+0000", 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5300000000000004e-07 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.575880+0000", 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4610000000000002e-06 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.195 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437261+0000", 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.86138053000000003 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437268+0000", 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6199999999999999e-06 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437339+0000", 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0621e-05 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.196 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.438028+0000", 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068918799999999998 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpRead(3.as1 58/56 ECSubRead(tid=40, to_read={3:5e52341f:::smithi01639990-38:head=0,262144,0}, subchunks={3:5e52341f:::smithi01639990-38:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:33.437230+0000", 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 38.711034032000001, 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.081032428000000004, 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.197 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437230+0000", 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437230+0000", 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.198 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437230+0000", 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6100000000000002e-07 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.437239+0000", 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2690000000000005e-06 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.517631+0000", 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.080391539999999997 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.199 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.517655+0000", 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4771999999999999e-05 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.517708+0000", 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2478999999999997e-05 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:33.518262+0000", 2024-09-16T04:09:12.200 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055410699999999997 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:522 3.14s0 3:2d08ec72:::smithi01639990-14:head [copy-from ver 47] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:37.619670+0000", 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.528593964999999, 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.32033554400000003, 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 522 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619670+0000", 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619670+0000", 2024-09-16T04:09:12.202 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619671+0000", 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5680000000000001e-06 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619677+0000", 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5160000000000002e-06 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619688+0000", 2024-09-16T04:09:12.203 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1276e-05 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619694+0000", 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5899999999999998e-06 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619749+0000", 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5640000000000003e-05 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.204 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619839+0000", 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9592000000000006e-05 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.643252+0000", 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.023413481999999999 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.647039+0000", 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037869969999999998 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.205 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.939885+0000", 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.29284586099999999 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.940005+0000", 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000120022 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.206 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:523 3.14s0 3:2d08ec72:::smithi01639990-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:37.619712+0000", 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.528551870999998, 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.026455427, 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 523 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.207 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619712+0000", 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619712+0000", 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.208 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619712+0000", 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8299999999999997e-07 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619717+0000", 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.707e-06 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619724+0000", 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0260000000000002e-06 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.209 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.619730+0000", 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6579999999999999e-06 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.620023+0000", 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00029293599999999999 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.620066+0000", 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3327e-05 2024-09-16T04:09:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.646035+0000", 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.025969142000000001 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.646100+0000", 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4986000000000004e-05 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.646167+0000", 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7162000000000005e-05 2024-09-16T04:09:12.211 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.14s0 62/48 ECSubWriteReply(tid=54, last_complete=62'37, committed=1, applied=1))", 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:37.649035+0000", 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.499228973999998, 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.29138913900000002, 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.212 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.649035+0000", 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.649035+0000", 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.649036+0000", 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6300000000000004e-07 2024-09-16T04:09:12.213 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.649041+0000", 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5060000000000003e-06 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.939461+0000", 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.290420077 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.939477+0000", 2024-09-16T04:09:12.214 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5956000000000001e-05 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.939647+0000", 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00017009800000000001 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:37.940424+0000", 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00077663899999999998 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.215 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.1s1 63/56 ECSubWrite(tid=110, reqid=osd.0.0:110, at_version=63'19, trim_to=0'0, roll_forward_to=62'17))", 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:38.513701+0000", 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 33.634563014999998, 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010654208, 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513701+0000", 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.216 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513701+0000", 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513701+0000", 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.35e-07 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.217 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513733+0000", 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2359e-05 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513769+0000", 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5868999999999998e-05 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.513783+0000", 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3995999999999999e-05 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514472+0000", 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00068909500000000003 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.514485+0000", 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2923e-05 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.524310+0000", 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098244760000000004 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.219 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.524355+0000", 2024-09-16T04:09:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5255e-05 2024-09-16T04:09:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.220 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.bs2 68/56 ECSubWrite(tid=329, reqid=osd.3.0:329, at_version=68'68, trim_to=0'0, roll_forward_to=67'62))", 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.541640+0000", 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.606624017000001, 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010394667999999999, 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541640+0000", 2024-09-16T04:09:12.221 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541640+0000", 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541640+0000", 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.75e-07 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541669+0000", 2024-09-16T04:09:12.222 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8578e-05 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541755+0000", 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5779999999999995e-05 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.541764+0000", 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3780000000000007e-06 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.223 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545213+0000", 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034488869999999999 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.545223+0000", 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0581000000000001e-05 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551990+0000", 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0067669560000000002 2024-09-16T04:09:12.224 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.552034+0000", 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3933000000000001e-05 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.17s7 68/56 ECSubWrite(tid=144, reqid=client.4440.0:650, at_version=68'65, trim_to=0'0, roll_forward_to=68'61))", 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.722856+0000", 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.425407428, 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.052442728000000001, 2024-09-16T04:09:12.225 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.722856+0000", 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.722856+0000", 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.226 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.722857+0000", 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1200000000000003e-07 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.722871+0000", 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4571e-05 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.774611+0000", 2024-09-16T04:09:12.227 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.051739988000000001 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.774617+0000", 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0800000000000002e-06 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.774665+0000", 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7555999999999997e-05 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.228 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.774671+0000", 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5809999999999996e-06 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.775270+0000", 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00059979300000000005 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.229 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.775299+0000", 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8547e-05 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.19s2 87/56 ECSubWrite(tid=191, reqid=client.4440.0:862, at_version=87'51, trim_to=0'0, roll_forward_to=67'48))", 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:03.021970+0000", 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.1262936949999993, 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.097252570999999996, 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.230 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021970+0000", 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021970+0000", 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.231 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021971+0000", 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2999999999999999e-07 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021999+0000", 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7888000000000001e-05 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.118478+0000", 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.096479121000000001 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.232 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.118484+0000", 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7089999999999999e-06 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.118565+0000", 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1089999999999998e-05 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.118576+0000", 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1489e-05 2024-09-16T04:09:12.233 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119194+0000", 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00061798300000000001 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119222+0000", 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8360999999999998e-05 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.234 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:919 3.3s0 3:c05d4807:::smithi01639990-31:head [copy-from ver 37] snapc 22=[22,1d,1c,15] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:08.482316+0000", 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.6659472019999999, 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.046789757000000001, 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 919 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.235 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482316+0000", 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482316+0000", 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482318+0000", 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.778e-06 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.236 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482327+0000", 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3790000000000006e-06 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482340+0000", 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3898e-05 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482362+0000", 2024-09-16T04:09:12.237 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1724000000000001e-05 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482516+0000", 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000154279 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482625+0000", 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108708 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.238 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.503389+0000", 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.020763613 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.506855+0000", 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034662249999999999 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.529040+0000", 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.022184953 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.239 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.529106+0000", 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6199999999999996e-05 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:920 3.3s0 3:c05d4807:::smithi01639990-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:08.482486+0000", 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 3.665777732, 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.024081551, 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.240 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 920 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482486+0000", 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.241 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482486+0000", 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482487+0000", 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7400000000000002e-07 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482496+0000", 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8959999999999992e-06 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.242 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482506+0000", 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0778000000000001e-05 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482514+0000", 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.977e-06 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482777+0000", 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00026276700000000001 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.243 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.482840+0000", 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2862999999999998e-05 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.506356+0000", 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.023515824000000001 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.506465+0000", 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109453 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.244 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.506567+0000", 2024-09-16T04:09:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010221900000000001 2024-09-16T04:09:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.245 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.245 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:12.247 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T04:09:12.247 INFO:tasks.ceph.osd.10.smithi194.stderr:2024-09-16T04:09:12.245+0000 7f37db404640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33295) UID: 0 2024-09-16T04:09:12.347 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-16T04:09:12.346+0000 7f46f9ac2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33070) UID: 0 2024-09-16T04:09:12.351 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:12.351 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:12.351 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:09:12.351 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:12.360 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T04:09:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2024-09-16T04:09:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'unique_pool_0' removed 2024-09-16T04:09:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: pool of 3.11 is pending removal 2024-09-16T04:09:12.448 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-16T04:09:12.446+0000 7fea03a8c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 47878) UID: 0 2024-09-16T04:09:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T04:09:12.455 INFO:tasks.thrashosds.thrasher:Failed to rm-pg-upmap-items, ignoring 2024-09-16T04:09:12.466 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T04:09:12.477 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T04:09:12.477 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:12.478 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:12.478 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:12.478 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:12.478 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:12.484 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T04:09:12.593 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 42/27 ECSubWriteReply(tid=88, last_complete=42'39, committed=1, applied=1))", 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 55.143183430000001, 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49630058100000002, 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.594 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448446+0000", 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448447+0000", 2024-09-16T04:09:12.595 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4799999999999997e-07 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.448451+0000", 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3030000000000001e-06 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944646+0000", 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.49619510999999999 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944655+0000", 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8480000000000007e-06 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944717+0000", 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2065999999999998e-05 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.944747+0000", 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9505999999999998e-05 2024-09-16T04:09:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDScrubReserve(3.0s7,QREQUEST,e:47,from:11(0),reservation_nonce:4)", 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 49.564461616000003, 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17961150300000001, 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:12.598 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027168+0000", 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027169+0000", 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8700000000000004e-07 2024-09-16T04:09:12.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.027173+0000", 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.1300000000000003e-06 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206636+0000", 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17946284700000001 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.600 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206644+0000", 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9650000000000008e-06 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206712+0000", 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8215999999999999e-05 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206735+0000", 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2789000000000002e-05 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.601 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:23.206779+0000", 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4669000000000002e-05 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:416 3.cs0 3:38c68ed3:::smithi01639990-42:head [append 0~294912 in=294912b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.101006908000002, 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16418209, 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:12.602 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 416 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490623+0000", 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490624+0000", 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3069999999999999e-06 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490921+0000", 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00029678400000000002 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.604 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490932+0000", 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1239e-05 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490938+0000", 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0379999999999996e-06 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490980+0000", 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2401999999999998e-05 2024-09-16T04:09:12.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491037+0000", 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6768000000000003e-05 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491591+0000", 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055345000000000004 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492749+0000", 2024-09-16T04:09:12.606 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0011580970000000001 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654745+0000", 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16199656800000001 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654805+0000", 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.9437e-05 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:417 3.cs0 3:38c68ed3:::smithi01639990-42:head [setxattr _header (57) in=64b] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.100676888999999, 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67294876800000003, 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.608 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 417 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.609 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490953+0000", 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2699999999999999e-07 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490956+0000", 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0130000000000002e-06 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490961+0000", 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1510000000000003e-06 2024-09-16T04:09:12.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490966+0000", 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7960000000000003e-06 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.491934+0000", 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00096816399999999998 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492020+0000", 2024-09-16T04:09:12.611 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5827000000000003e-05 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492479+0000", 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00045912299999999997 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492973+0000", 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00049418400000000005 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:12.612 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163802+0000", 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67082863599999998 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163901+0000", 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9246999999999994e-05 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:418 3.cs0 3:38c68ed3:::smithi01639990-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:12.613 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.100655709999998, 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67426341400000001, 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 418 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.614 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490974+0000", 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8000000000000001e-07 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.615 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490976+0000", 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.973e-06 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490983+0000", 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4699999999999999e-06 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.490988+0000", 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5509999999999999e-06 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.616 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493081+0000", 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020931029999999998 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493165+0000", 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3988999999999998e-05 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164330+0000", 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67116498999999996 2024-09-16T04:09:12.617 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164423+0000", 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000004e-05 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.165237+0000", 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00081446800000000005 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.618 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 52/27 ECSubWriteReply(tid=99, last_complete=52'26, committed=1, applied=1))", 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.098714018999999, 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16189895200000001, 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492915+0000", 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492916+0000", 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1699999999999998e-07 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.492919+0000", 2024-09-16T04:09:12.620 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6970000000000002e-06 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654619+0000", 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.16170007 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654628+0000", 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9080000000000001e-06 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.621 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654685+0000", 2024-09-16T04:09:12.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7349999999999998e-05 2024-09-16T04:09:12.622 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.622 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.622 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.622 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.654814+0000", 2024-09-16T04:09:12.622 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00012941 2024-09-16T04:09:12.622 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.622 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.622 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.623 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.623 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.624 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 52/27 ECSubWriteReply(tid=100, last_complete=52'27, committed=1, applied=1))", 2024-09-16T04:09:12.624 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:09:12.624 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 45.098271947000001, 2024-09-16T04:09:12.624 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67093597800000004, 2024-09-16T04:09:12.624 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.624 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493357+0000", 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.625 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493358+0000", 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6800000000000001e-07 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:27.493360+0000", 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.2019999999999998e-06 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163618+0000", 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67025769000000002 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163644+0000", 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6089999999999999e-05 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.163713+0000", 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9326999999999995e-05 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.164293+0000", 2024-09-16T04:09:12.627 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00058030099999999997 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s4 63/56 ECSubWrite(tid=277, reqid=osd.3.0:277, at_version=63'79, trim_to=0'0, roll_forward_to=62'77))", 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 34.080282506000003, 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.70853546499999998, 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.628 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511347+0000", 2024-09-16T04:09:12.629 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4099999999999999e-07 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:38.511375+0000", 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7755000000000001e-05 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219030+0000", 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.70765444700000002 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.630 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219046+0000", 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.6152999999999999e-05 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219182+0000", 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013656599999999999 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219190+0000", 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1079999999999993e-06 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.631 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219848+0000", 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065702000000000002 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:39.219882+0000", 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4875000000000001e-05 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.632 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 67/56 ECSubWriteReply(tid=146, last_complete=67'24, committed=1, applied=1))", 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:42.548179+0000", 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 30.043450199999999, 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.18309741900000001, 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548179+0000", 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.633 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548179+0000", 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548180+0000", 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4199999999999995e-07 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.548184+0000", 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4700000000000004e-06 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.634 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.730637+0000", 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.182453118 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.730647+0000", 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0650000000000005e-06 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.730717+0000", 2024-09-16T04:09:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0551999999999998e-05 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:42.731277+0000", 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00055957200000000004 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds1 81/56 ECSubWrite(tid=221, reqid=osd.7.0:221, at_version=81'36, trim_to=0'0, roll_forward_to=80'34))", 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:56.485832+0000", 2024-09-16T04:09:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.105797209999999, 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67555770299999995, 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485832+0000", 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485832+0000", 2024-09-16T04:09:12.637 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485833+0000", 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3299999999999998e-07 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.485854+0000", 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1855999999999999e-05 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.638 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160608+0000", 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.67475406599999999 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160614+0000", 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.789e-06 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160698+0000", 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3311000000000002e-05 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.639 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.160705+0000", 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.161360+0000", 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065466099999999996 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.161390+0000", 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0108999999999999e-05 2024-09-16T04:09:12.640 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:780 3.cs0 3:38c68ed3:::smithi01639990-42:head [copy-from ver 22] snapc 1d=[1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.488449+0000", 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.103180894999999, 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.022970059000000001, 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.641 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 780 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488449+0000", 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488449+0000", 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488451+0000", 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7609999999999999e-06 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488464+0000", 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2547e-05 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488485+0000", 2024-09-16T04:09:12.643 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0789999999999999e-05 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488497+0000", 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1919000000000001e-05 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488579+0000", 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2398999999999995e-05 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.644 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488588+0000", 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7309999999999999e-06 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490560+0000", 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0019719410000000001 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490628+0000", 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8275000000000006e-05 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.645 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.504399+0000", 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013771097 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506432+0000", 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020333460000000001 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.511365+0000", 2024-09-16T04:09:12.646 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049331410000000003 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.511419+0000", 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3112000000000002e-05 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:781 3.cs0 3:38c68ed3:::smithi01639990-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.488524+0000", 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 15.103105293, 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017905164000000001, 2024-09-16T04:09:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 781 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488524+0000", 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.648 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488524+0000", 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488525+0000", 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6400000000000002e-07 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.649 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488528+0000", 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2440000000000002e-06 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488537+0000", 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6209999999999993e-06 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488542+0000", 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6890000000000001e-06 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.650 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488657+0000", 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011471799999999999 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.488663+0000", 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1330000000000004e-06 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490738+0000", 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0020745709999999999 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.651 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.490765+0000", 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6897999999999999e-05 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506037+0000", 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015272819 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506239+0000", 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00020141000000000001 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.652 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.506429+0000", 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000190497 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:862 3.19s0 3:9fd10cc9:::smithi01639990-50:head [setxattr _header (57) in=7b] snapc 20=[1f,1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:03.019734+0000", 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.571895391, 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099704702000000006, 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.653 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 862 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019734+0000", 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.654 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019734+0000", 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019735+0000", 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019755+0000", 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0129999999999999e-05 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.655 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019761+0000", 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6680000000000004e-06 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019767+0000", 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3299999999999998e-06 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021558+0000", 2024-09-16T04:09:12.656 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001791656 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021628+0000", 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9875000000000004e-05 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.021980+0000", 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00035168999999999999 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.657 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.022837+0000", 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00085693300000000004 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119380+0000", 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.096543290000000004 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119439+0000", 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8539999999999999e-05 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.658 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:863 3.19s0 3:9fd10cc9:::smithi01639990-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:03.019795+0000", 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 9.5718348150000008, 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.10150271399999999, 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 863 2024-09-16T04:09:12.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019795+0000", 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019795+0000", 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019795+0000", 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5499999999999998e-07 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019799+0000", 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6940000000000001e-06 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019806+0000", 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6650000000000003e-06 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.661 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.019813+0000", 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9369999999999998e-06 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.022297+0000", 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0024831499999999999 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.022386+0000", 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.9072000000000004e-05 2024-09-16T04:09:12.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.119960+0000", 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.097574004000000006 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.120038+0000", 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8032000000000001e-05 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:03.121297+0000", 2024-09-16T04:09:12.663 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012597050000000001 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds1 92/56 ECSubWrite(tid=234, reqid=osd.7.0:234, at_version=92'44, trim_to=0'0, roll_forward_to=89'42))", 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:07.450700+0000", 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.1409290439999999, 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019189081, 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.664 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.450700+0000", 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.450700+0000", 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.450701+0000", 2024-09-16T04:09:12.665 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.450718+0000", 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7430000000000001e-05 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.450795+0000", 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7082999999999994e-05 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.666 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.450801+0000", 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7209999999999999e-06 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.450848+0000", 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7135999999999997e-05 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.450858+0000", 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.5980000000000003e-06 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.667 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.469866+0000", 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019008437999999999 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.469889+0000", 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3439e-05 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.668 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.1s4 92/56 ECSubWrite(tid=226, reqid=osd.0.0:226, at_version=92'36, trim_to=0'0, roll_forward_to=89'34))", 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:07.452313+0000", 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.1393169040000002, 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017144005, 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.452313+0000", 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.669 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.452313+0000", 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.452313+0000", 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7000000000000001e-07 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.452334+0000", 2024-09-16T04:09:12.670 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0842000000000001e-05 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.452541+0000", 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000207071 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.452551+0000", 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0044000000000001e-05 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.671 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.456095+0000", 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035443200000000001 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.456104+0000", 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7190000000000007e-06 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.469425+0000", 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013321515000000001 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.672 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.469457+0000", 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1223999999999998e-05 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 92/56 ECSubWrite(tid=188, reqid=osd.8.0:188, at_version=92'111, trim_to=0'0, roll_forward_to=91'109))", 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:07.455979+0000", 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.1356504789999997, 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019103901, 2024-09-16T04:09:12.673 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.455979+0000", 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.455979+0000", 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.455979+0000", 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.04e-07 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.456002+0000", 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3147999999999999e-05 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.675 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.456067+0000", 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4731000000000001e-05 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.456074+0000", 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.635e-06 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.457485+0000", 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001411645 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.676 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.457494+0000", 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0909999999999999e-06 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.475057+0000", 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017562286999999999 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.475083+0000", 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.616e-05 2024-09-16T04:09:12.677 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.9s4 92/56 ECSubWrite(tid=189, reqid=osd.8.0:189, at_version=92'113, trim_to=0'0, roll_forward_to=91'109))", 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:07.459656+0000", 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.1319733540000003, 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017987748000000001, 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.678 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.459656+0000", 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.459656+0000", 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.459657+0000", 2024-09-16T04:09:12.679 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1299999999999999e-07 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.459686+0000", 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8717e-05 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.459703+0000", 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.7024000000000002e-05 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.680 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.459715+0000", 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2184000000000001e-05 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.469493+0000", 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097780980000000007 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.469506+0000", 2024-09-16T04:09:12.681 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3084999999999999e-05 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.477608+0000", 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0081020490000000001 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.477644+0000", 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5877999999999998e-05 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.682 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:901 3.16s0 3:6b28745a:::smithi01639990-19:head [setxattr _header (57) in=7b] snapc 22=[22,1f,1d,1c,15] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:07.460553+0000", 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.1310763719999999, 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.020463274, 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 901 2024-09-16T04:09:12.683 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460553+0000", 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460553+0000", 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.684 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460554+0000", 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5000000000000002e-07 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460557+0000", 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5559999999999999e-06 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460566+0000", 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2930000000000001e-06 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.685 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460572+0000", 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0059999999999998e-06 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.465714+0000", 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0051422940000000004 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.465823+0000", 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010949 2024-09-16T04:09:12.686 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.466693+0000", 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00086934000000000002 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.467715+0000", 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001022511 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:12.687 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.480929+0000", 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.013214052 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.481016+0000", 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6982000000000002e-05 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:902 3.16s0 3:6b28745a:::smithi01639990-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-16T04:09:12.688 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:07.460581+0000", 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 5.1310487030000003, 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.030498229000000002, 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 902 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:12.689 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460581+0000", 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460581+0000", 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460581+0000", 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6100000000000001e-07 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.690 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460584+0000", 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5809999999999999e-06 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460591+0000", 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.5560000000000002e-06 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.460604+0000", 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2384999999999999e-05 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.691 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.466992+0000", 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0063882929999999997 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.467082+0000", 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0488999999999994e-05 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.481389+0000", 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014306223999999999 2024-09-16T04:09:12.692 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.481488+0000", 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9353999999999998e-05 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:07.491079+0000", 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095908859999999999 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:12.693 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:12.694 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:12.694 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T04:09:12.798 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:12.798 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:12.798 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:09:12.798 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:12.807 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T04:09:12.909 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:12.910 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:12.910 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:12.910 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:12.910 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:12.918 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T04:09:13.023 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:13.023 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:13.023 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:13.023 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:13.023 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.023 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/27 ECSubWrite(tid=158, reqid=client.4440.0:346, at_version=42'31, trim_to=0'0, roll_forward_to=41'26))", 2024-09-16T04:09:13.023 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:13.023 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 55.570399018000003, 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0091491550000000008, 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.024 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.449940+0000", 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5800000000000001e-07 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450512+0000", 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00057160000000000002 2024-09-16T04:09:13.025 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450541+0000", 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8835000000000001e-05 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450549+0000", 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3080000000000008e-06 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450607+0000", 2024-09-16T04:09:13.026 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7763999999999998e-05 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.450616+0000", 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0860000000000008e-06 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459045+0000", 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0084290709999999998 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.027 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:17.459089+0000", 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4233000000000002e-05 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:469 3.as0 3:5e52341f:::smithi01639990-38:head [append 0~491520] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.467566054000002, 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098323030000000006, 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.028 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 469 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.029 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552773+0000", 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.552774+0000", 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1999999999999999e-06 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553284+0000", 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00050997900000000003 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.030 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553297+0000", 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2723e-05 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553306+0000", 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5969999999999992e-06 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.031 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553363+0000", 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6922999999999997e-05 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.553436+0000", 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3090000000000007e-05 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.554087+0000", 2024-09-16T04:09:13.032 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00065173499999999997 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.558220+0000", 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0041326330000000001 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562517+0000", 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042965859999999998 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.033 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.562605+0000", 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8837000000000003e-05 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:474 3.as0 3:5e52341f:::smithi01639990-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 44.460234092999997, 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0097686969999999998, 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:13.034 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 474 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560105+0000", 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.035 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560106+0000", 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.63e-07 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560109+0000", 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.303e-06 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.036 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560116+0000", 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3860000000000001e-06 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.560122+0000", 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.4450000000000004e-06 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563686+0000", 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0035639740000000001 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.037 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.563776+0000", 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0278000000000005e-05 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568854+0000", 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0050781680000000001 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.568970+0000", 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011579899999999999 2024-09-16T04:09:13.038 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:28.569874+0000", 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00090388100000000004 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.582602397999999, 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015912599999999999, 2024-09-16T04:09:13.039 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.040 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437737+0000", 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3300000000000001e-07 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437757+0000", 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9735e-05 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437829+0000", 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1605000000000003e-05 2024-09-16T04:09:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437838+0000", 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.0610000000000002e-06 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442352+0000", 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045136799999999999 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442364+0000", 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2642e-05 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.042 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453618+0000", 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011254128 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453649+0000", 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1016000000000003e-05 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.043 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s5 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.581563789, 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014897297, 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.044 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438775+0000", 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438776+0000", 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3099999999999997e-07 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438797+0000", 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0596000000000001e-05 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.045 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438836+0000", 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.9684000000000002e-05 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.438845+0000", 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.6479999999999992e-06 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442677+0000", 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0038317239999999999 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.046 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442685+0000", 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453654+0000", 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.010968637 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453673+0000", 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8967999999999999e-05 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.047 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=57'31, trim_to=0'0, roll_forward_to=53'29))", 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.579573840000002, 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012700487999999999, 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.048 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440765+0000", 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8900000000000001e-07 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.049 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440785+0000", 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.9850000000000001e-05 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445485+0000", 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046992329999999997 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445495+0000", 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0506000000000001e-05 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.050 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445668+0000", 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000172905 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.445679+0000", 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453438+0000", 2024-09-16T04:09:13.051 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0077588930000000002 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.453466+0000", 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8175e-05 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 57/27 ECSubWrite(tid=110, reqid=osd.5.0:109, at_version=0'0, trim_to=0'0, roll_forward_to=57'31))", 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:13.052 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.555769388000002, 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011518758, 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:13.053 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464570+0000", 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8e-07 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.464577+0000", 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9940000000000003e-06 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472531+0000", 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0079544330000000003 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472538+0000", 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7789999999999998e-06 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472599+0000", 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0841999999999998e-05 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.055 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.472610+0000", 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1185999999999999e-05 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476060+0000", 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0034496650000000002 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.476089+0000", 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8679000000000001e-05 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.056 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=37, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.532266585999999, 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.039596059000000003, 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.057 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488073+0000", 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1e-07 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.058 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.488075+0000", 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8789999999999999e-06 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526262+0000", 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.038187556999999997 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526270+0000", 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.3209999999999999e-06 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.059 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.526424+0000", 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015469 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.527669+0000", 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0012444019999999999 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.060 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.as0 57/56 ECSubWriteReply(tid=38, last_complete=33'10, committed=1, applied=1))", 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 40.489774308999998, 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028484058, 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.061 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530565+0000", 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530566+0000", 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.85e-07 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.530570+0000", 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7319999999999998e-06 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.062 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558866+0000", 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.028295515 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558875+0000", 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7849999999999997e-06 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.558966+0000", 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.1611000000000001e-05 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.559049+0000", 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.263e-05 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.cs2 69/27 ECSubWrite(tid=153, reqid=osd.5.0:153, at_version=69'41, trim_to=0'0, roll_forward_to=68'39))", 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.481020891, 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0089824069999999995, 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.064 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.065 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539318+0000", 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539343+0000", 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4399000000000001e-05 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539420+0000", 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7167000000000002e-05 2024-09-16T04:09:13.066 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539428+0000", 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0679999999999997e-06 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539506+0000", 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7836000000000005e-05 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539518+0000", 2024-09-16T04:09:13.067 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1627000000000001e-05 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548257+0000", 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0087393379999999993 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548301+0000", 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3798999999999999e-05 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.068 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.14s5 69/48 ECSubWrite(tid=71, reqid=osd.4.0:71, at_version=69'49, trim_to=0'0, roll_forward_to=68'45))", 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.480976641000002, 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0093116210000000008, 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539363+0000", 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0999999999999998e-07 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539388+0000", 2024-09-16T04:09:13.070 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4769999999999998e-05 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539439+0000", 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1149000000000001e-05 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539450+0000", 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0557000000000001e-05 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.071 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539608+0000", 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000158679 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539620+0000", 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1260000000000001e-05 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548634+0000", 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.009014256 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.072 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.548674+0000", 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.0240000000000001e-05 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.ds3 69/56 ECSubWrite(tid=201, reqid=client.4440.0:654, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.479774202000002, 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098306729999999998, 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.073 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.074 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540565+0000", 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6600000000000002e-07 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540718+0000", 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015259 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540851+0000", 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00013328999999999999 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.075 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540860+0000", 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7909999999999993e-06 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541117+0000", 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025658799999999998 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541127+0000", 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0834e-05 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.076 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.550371+0000", 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092436089999999999 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.550396+0000", 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4505e-05 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.12s6 69/56 ECSubWrite(tid=147, reqid=osd.0.0:147, at_version=69'84, trim_to=0'0, roll_forward_to=68'82))", 2024-09-16T04:09:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.542608+0000", 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.477730730000001, 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021420679000000001, 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542608+0000", 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.078 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542608+0000", 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542609+0000", 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.15e-07 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542633+0000", 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4542999999999999e-05 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.079 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542748+0000", 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000114911 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542756+0000", 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542862+0000", 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000105689 2024-09-16T04:09:13.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.542873+0000", 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0575e-05 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.563997+0000", 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021124446000000002 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564029+0000", 2024-09-16T04:09:13.081 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2101000000000003e-05 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.12s6 69/56 ECSubWrite(tid=149, reqid=osd.0.0:149, at_version=69'86, trim_to=0'0, roll_forward_to=68'82))", 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.544392+0000", 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.475947136999999, 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019668749999999999, 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.082 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544392+0000", 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544392+0000", 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544393+0000", 2024-09-16T04:09:13.083 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.195e-06 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544420+0000", 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6635000000000002e-05 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544463+0000", 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3479000000000002e-05 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.084 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.544472+0000", 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.8880000000000003e-06 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.551267+0000", 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00679526 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.551277+0000", 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2760000000000006e-06 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.085 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564037+0000", 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012760258 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564061+0000", 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3759000000000001e-05 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.086 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 69/56 ECSubWrite(tid=340, reqid=osd.3.0:340, at_version=69'98, trim_to=0'0, roll_forward_to=68'96))", 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.465930409999999, 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0096797529999999993, 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554409+0000", 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554437+0000", 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.7382e-05 2024-09-16T04:09:13.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554469+0000", 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.2073000000000003e-05 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554480+0000", 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1358999999999999e-05 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554550+0000", 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9808999999999995e-05 2024-09-16T04:09:13.089 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.554564+0000", 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3634e-05 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564066+0000", 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0095028419999999992 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.564088+0000", 2024-09-16T04:09:13.090 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1996000000000001e-05 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.2s2 74/56 ECSubWrite(tid=376, reqid=osd.3.0:376, at_version=74'104, trim_to=0'0, roll_forward_to=70'102))", 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:49.453728+0000", 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 23.566610656999998, 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0092035429999999998, 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.091 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453728+0000", 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453728+0000", 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453729+0000", 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1e-07 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.092 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453787+0000", 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8541000000000001e-05 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453868+0000", 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1199000000000006e-05 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453875+0000", 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.8310000000000003e-06 2024-09-16T04:09:13.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453979+0000", 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000103288 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.453988+0000", 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.533e-06 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.094 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.462890+0000", 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0089015139999999993 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:49.462932+0000", 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2426999999999999e-05 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:736 3.as0 3:5e52341f:::smithi01639990-38:head [copy-from ver 69] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439036+0000", 2024-09-16T04:09:13.095 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.581303174999999, 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.030166564999999999, 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 736 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.096 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439036+0000", 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439036+0000", 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439038+0000", 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4339999999999999e-06 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.097 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439050+0000", 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1418e-05 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439070+0000", 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0251e-05 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439080+0000", 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4599999999999992e-06 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.098 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439154+0000", 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4820999999999994e-05 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439162+0000", 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6969999999999993e-06 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.443704+0000", 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0045422580000000004 2024-09-16T04:09:13.099 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.443816+0000", 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011184900000000001 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.461800+0000", 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017984236000000001 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.465505+0000", 2024-09-16T04:09:13.100 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0037041209999999999 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.469122+0000", 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0036175270000000002 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.469203+0000", 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.0493000000000003e-05 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:737 3.as0 3:5e52341f:::smithi01639990-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439252+0000", 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 20.581086995, 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.024601304000000001, 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 737 2024-09-16T04:09:13.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439252+0000", 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439252+0000", 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.103 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439253+0000", 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7899999999999998e-07 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439259+0000", 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3609999999999997e-06 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439269+0000", 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8430000000000005e-06 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.104 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439276+0000", 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4649999999999997e-06 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439328+0000", 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.1063999999999998e-05 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439334+0000", 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7649999999999996e-06 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.105 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.443976+0000", 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0046418049999999997 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.444023+0000", 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6872000000000001e-05 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.463639+0000", 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019616097999999998 2024-09-16T04:09:13.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.463746+0000", 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010683600000000001 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.463853+0000", 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00010751600000000001 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.107 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:930 3.as0 3:5e52341f:::smithi01639990-38:head [stat,delete] snapc 22=[22,1d,1c,15] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:08.485019+0000", 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.5353205399999998, 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011505372, 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 930 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.108 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.485019+0000", 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.485019+0000", 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.485020+0000", 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1790000000000001e-06 2024-09-16T04:09:13.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.485024+0000", 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.4819999999999996e-06 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.485036+0000", 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1579e-05 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.110 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.485042+0000", 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6690000000000003e-06 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.485087+0000", 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.5939000000000002e-05 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.485157+0000", 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9647000000000006e-05 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.111 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.485679+0000", 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00052226099999999999 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.486601+0000", 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00092209699999999998 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.496465+0000", 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098638590000000009 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.112 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.496524+0000", 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8659999999999997e-05 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "rep_scrubmap(3.as0 e56 from shard 4(1))", 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:11.943030+0000", 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 1.0773092150000001, 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.55134018399999996, 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.113 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "queued for pg", 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:11.943030+0000", 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:11.943030+0000", 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.114 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:11.943031+0000", 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1850000000000001e-06 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:11.943179+0000", 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014826800000000001 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:12.494205+0000", 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.55102523299999995 2024-09-16T04:09:13.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:12.494251+0000", 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6008999999999999e-05 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:12.494370+0000", 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011948899999999999 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.116 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.117 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:13.118 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T04:09:13.221 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:13.221 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:13.221 INFO:teuthology.orchestra.run.smithi174.stdout: "num_ops": 0 2024-09-16T04:09:13.221 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:13.230 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T04:09:13.331 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:13.331 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [], 2024-09-16T04:09:13.332 INFO:teuthology.orchestra.run.smithi174.stdout: "complaint_time": 30, 2024-09-16T04:09:13.332 INFO:teuthology.orchestra.run.smithi174.stdout: "num_blocked_ops": 0 2024-09-16T04:09:13.332 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:13.340 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T04:09:13.443 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-16T04:09:13.443 INFO:teuthology.orchestra.run.smithi174.stdout: "size": 20, 2024-09-16T04:09:13.443 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 600, 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "ops": [ 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 83.500146783999995, 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17988575500000001, 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:13.444 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 1 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940723+0000", 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.445 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940724+0000", 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2359999999999999e-06 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940737+0000", 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3103e-05 2024-09-16T04:09:13.446 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940746+0000", 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2490000000000007e-06 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940755+0000", 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.1580000000000005e-06 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.447 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940803+0000", 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.8077000000000001e-05 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for peered", 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:49.940815+0000", 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2619999999999999e-05 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119223+0000", 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.17840792899999999 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119348+0000", 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000124828 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.119515+0000", 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016711699999999999 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120350+0000", 2024-09-16T04:09:13.449 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00083520199999999995 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120543+0000", 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00019279 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120579+0000", 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.5976999999999998e-05 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.120609+0000", 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.9468999999999999e-05 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.517742541000004, 2024-09-16T04:09:13.451 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85974535399999996, 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 37 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.452 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923127+0000", 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.29e-07 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.453 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923136+0000", 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2309999999999996e-06 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923146+0000", 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0737000000000001e-05 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.923150+0000", 2024-09-16T04:09:13.454 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3110000000000001e-06 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930091+0000", 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0069411539999999997 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930167+0000", 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.6734000000000001e-05 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.455 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.930250+0000", 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.2448999999999996e-05 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.931208+0000", 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00095810800000000005 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782784+0000", 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.85157554000000002 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.456 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782822+0000", 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8282999999999999e-05 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782872+0000", 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0578000000000003e-05 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_repop_reply(client.4192.0:37 1.0 e20/19 ondisk, result = 0)", 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 82.507380034999997, 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84939863900000001, 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "started", 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.458 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933490+0000", 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.7499999999999999e-07 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.459 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:50.933497+0000", 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5649999999999998e-06 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782614+0000", 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.84911649300000003 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782624+0000", 2024-09-16T04:09:13.460 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0189e-05 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782768+0000", 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000144122 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782779+0000", 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0891e-05 2024-09-16T04:09:13.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:51.782888+0000", 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000109504 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4192.0:124 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 81.010791983000004, 2024-09-16T04:09:13.462 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.021096521, 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4192", 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.174:0/654099619", 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 124 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:09:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430078+0000", 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430080+0000", 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5890000000000001e-06 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.464 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430091+0000", 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0890000000000001e-05 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430109+0000", 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.8199e-05 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430119+0000", 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.8830000000000001e-06 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.465 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430231+0000", 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000112187 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430340+0000", 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000108544 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting for subops from 11", 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.430504+0000", 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016360100000000001 2024-09-16T04:09:13.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op_commit", 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.431146+0000", 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00064253400000000005 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_commit_rec", 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451107+0000", 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019960898000000001 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451144+0000", 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6671999999999997e-05 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.467 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:07:52.451174+0000", 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0524000000000001e-05 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDPGPushReply(3.ds0 32/31 [PushReplyOp(3:bc502819:::smithi01639990-5:head)])", 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 66.331894891000005, 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.57039183800000004, 2024-09-16T04:09:13.468 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.469 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108975+0000", 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5199999999999998e-07 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.108977+0000", 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0030000000000001e-06 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677629+0000", 2024-09-16T04:09:13.470 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.56865195499999999 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677636+0000", 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4889999999999998e-06 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.677693+0000", 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.6986e-05 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.471 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.679367+0000", 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0016731529999999999 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.6s3 32/27 ECSubWrite(tid=15, reqid=client.4440.0:181, at_version=0'0, trim_to=0'0, roll_forward_to=32'6))", 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 65.753960930999995, 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89654780700000003, 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.472 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.473 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686909+0000", 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.5799999999999999e-07 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:07.686918+0000", 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5869999999999993e-06 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582762+0000", 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.89584425400000001 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.474 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582783+0000", 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1182000000000001e-05 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582883+0000", 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.9229999999999997e-05 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.582890+0000", 2024-09-16T04:09:13.475 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.2819999999999999e-06 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583423+0000", 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00053294799999999999 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:08.583457+0000", 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.3665999999999999e-05 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.5s4 33/27 ECSubWrite(tid=39, reqid=osd.5.0:39, at_version=33'12, trim_to=0'0, roll_forward_to=32'10))", 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 64.337147786000003, 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012569337999999999, 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.477 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103722+0000", 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.73e-07 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.478 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.103762+0000", 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.964e-05 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104137+0000", 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00037533899999999998 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104146+0000", 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5129999999999996e-06 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.479 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104224+0000", 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.8712000000000001e-05 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.104238+0000", 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3899e-05 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116253+0000", 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012015179000000001 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.480 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.116291+0000", 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7883000000000003e-05 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 33/27 ECSubWrite(tid=78, reqid=client.4440.0:207, at_version=33'21, trim_to=0'0, roll_forward_to=32'17))", 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 64.335760371999996, 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012155859999999999, 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.481 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105109+0000", 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105110+0000", 2024-09-16T04:09:13.482 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105370+0000", 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00025967600000000003 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105408+0000", 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.8334999999999998e-05 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105415+0000", 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:09:13.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105467+0000", 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.2787000000000001e-05 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.105474+0000", 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1879999999999996e-06 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117219+0000", 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011744203 2024-09-16T04:09:13.484 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:09.117265+0000", 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.6440000000000003e-05 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/46 ECSubWrite(tid=189, reqid=client.4440.0:379, at_version=0'0, trim_to=0'0, roll_forward_to=47'56))", 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 50.976441342000001, 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.100081218, 2024-09-16T04:09:13.485 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464428+0000", 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.486 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464429+0000", 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.5499999999999998e-07 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.464437+0000", 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.3350000000000007e-06 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563655+0000", 2024-09-16T04:09:13.487 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.099217464000000005 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563664+0000", 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4439999999999997e-06 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563739+0000", 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.4957999999999996e-05 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.488 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.563751+0000", 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2272999999999999e-05 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564475+0000", 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00072406 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:22.564510+0000", 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.4128999999999998e-05 2024-09-16T04:09:13.489 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=80, reqid=osd.8.0:80, at_version=57'49, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 41.003571391999998, 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019153804, 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.490 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437298+0000", 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.491 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437299+0000", 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8100000000000003e-07 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437320+0000", 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1121999999999999e-05 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437388+0000", 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.7889e-05 2024-09-16T04:09:13.492 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.437397+0000", 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4390000000000006e-06 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441658+0000", 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0042607110000000004 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.493 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.441667+0000", 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4089999999999992e-06 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456416+0000", 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.014748493 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.456452+0000", 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.6159999999999999e-05 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.494 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.15s2 57/48 ECSubWrite(tid=81, reqid=osd.8.0:81, at_version=57'51, trim_to=0'0, roll_forward_to=53'47))", 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 41.000381681, 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017174697999999999, 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:09:13.495 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440488+0000", 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440489+0000", 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.015e-06 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.496 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440509+0000", 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.0338000000000001e-05 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440547+0000", 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7808000000000001e-05 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.440557+0000", 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.7820000000000005e-06 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.497 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442068+0000", 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0015112179999999999 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.442078+0000", 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4620000000000002e-06 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457639+0000", 2024-09-16T04:09:13.498 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.015560954 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:32.457663+0000", 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.4121000000000002e-05 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.as7 68/56 ECSubWrite(tid=57, reqid=osd.6.0:57, at_version=68'42, trim_to=0'0, roll_forward_to=64'40))", 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:13.499 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.906714429000001, 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.017528866000000001, 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.500 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534155+0000", 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.49e-07 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.534179+0000", 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3308000000000001e-05 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.501 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546651+0000", 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012471873 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546660+0000", 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2229999999999996e-06 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546730+0000", 2024-09-16T04:09:13.502 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.0123000000000006e-05 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.546741+0000", 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.0684e-05 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551646+0000", 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0049059180000000004 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.503 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.551684+0000", 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7588e-05 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.13s0 68/56 ECSubWriteReply(tid=200, last_complete=68'24, committed=1, applied=1))", 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 29.877702405000001, 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46574167999999999, 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.504 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563167+0000", 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.505 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563168+0000", 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4099999999999997e-07 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:43.563174+0000", 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.327e-06 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028683+0000", 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.46550948399999997 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.506 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028698+0000", 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.4576000000000001e-05 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028858+0000", 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00016062100000000001 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.028909+0000", 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.0730999999999999e-05 2024-09-16T04:09:13.507 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:654 3.ds0 3:bc502819:::smithi01639990-5:head [create,setxattr XXWdXV_kiPaajZX\\PgmqNaZqSVmab^hrtRYeagZr (1894) in=40b,setxattr NdZr`e\\^kXiksh]\\XocenQrhYc]c`[proOfo`Pd^ (33) in=40b,setxattr _`NqSdldlQbUNl^ZNRba\\naZhhdraiOgpafrZb[o (103) in=40b,setxattr iadimrq_[jOPUblhnVqTYqQNedg (1647) in=27b,setxattr `e\\rtQi]nmoP\\cm]oRYN`[]^qVUTsU^_oXm]h\\ec (6452) in=40b,setxattr ppZXc_dYi_WWXcUdeqcpck_ceUVa^XftgnfgdP[q (3576) in=40b,setxattr hehZsP\\ajqhe^nfrZjQmsnfWoqq\\e\\NV^rOaZYtj (1787) in=40b,setxattr a]Smbg`^k`emWabQRYcpr_lQj`eh`[r\\riqa_sfd (1015) in=40b,setxattr eqsaZkZ[`t^R^grhYOVrVsspVQQp_Zo\\loUV[\\U[ (982) in=40b,setxattr Rn`k`cSYQj`XOZdn_mSPrqQtoUpWTk[^[t (1647) in=34b,setxattr ]TtVabQgUTabfmqae[UVe`cNsqqYfdh]\\sVoeXQr (383) in=40b,setxattr iRn][N_NY\\ZhmYdQ[[ptTP_NW^Z]_roZnOpcmR_Y (19016) in=40b,setxattr T_b_WhnZnkaaYkNmmOUnaZa`dhRhQSshYQVQnVql (13472) in=40b,setxattr \\hZXh`dT^W[c\\XodZZ_d\\QO\\\\[PVWthO_WVOPSQd (1681) in=40b,setxattr ctSOqNY_dUqTOjgXYjqgslgNRRPoT\\\\OXPPWrtt_ (522) in=40b,setxattr f\\hTOf^sbRaYeUntSV^WhUo]qV`bSNermptfXs]Q (1645) in=40b,setxattr ohNTdfibmYs`eUTiQa^bcYW\\aarhT]YQ_haagqpf (1119) in=40b,setxattr qqUtZZ]ataUj_[pnh\\jVtU_dRpi^g[VPNeS`Y\\hk (699) in=40b,setxattr qsgedRfqdW]]cYbWOUgh^ZleaSQiteejr^afXbUj (525) in=40b,setxattr [XmPqpoSg`fWjdlmQ_qabof]_WVcPaY[]k[ardgj (594) in=40b] snapc 15=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 28.906691370000001, 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019213674, 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 654 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534178+0000", 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.509 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534181+0000", 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.5830000000000001e-06 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534398+0000", 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000216663 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.510 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534421+0000", 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.3714e-05 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534434+0000", 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3185e-05 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534504+0000", 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9665000000000005e-05 2024-09-16T04:09:13.511 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.534510+0000", 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.1709999999999999e-06 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.538902+0000", 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0043914660000000001 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.539044+0000", 2024-09-16T04:09:13.512 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00014215000000000001 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.540400+0000", 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0013559189999999999 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.541692+0000", 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.001291964 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.513 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553277+0000", 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.011585432999999999 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:44.553392+0000", 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011476099999999999 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.514 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:733 3.fs0 3:f7af954f:::smithi01639990-10:head [append 0~0] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.002059720999998, 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.016112728999999999, 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 733 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.515 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438810+0000", 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438813+0000", 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.6089999999999999e-06 2024-09-16T04:09:13.516 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438822+0000", 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.6199999999999994e-06 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438844+0000", 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1933000000000002e-05 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.517 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438856+0000", 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.1588e-05 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438966+0000", 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00011003899999999999 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "op must wait for map", 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438974+0000", 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.5019999999999992e-06 2024-09-16T04:09:13.518 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.446497+0000", 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0075231389999999999 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.446592+0000", 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.4358999999999996e-05 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.447539+0000", 2024-09-16T04:09:13.519 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00094693900000000005 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454766+0000", 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0072270520000000003 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454838+0000", 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1970000000000004e-05 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.520 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.454923+0000", 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 8.4979000000000001e-05 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:734 3.fs0 3:f7af954f:::smithi01639990-10:head [rollback 1a] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.001985668, 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019165187, 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.521 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 734 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:13.522 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438884+0000", 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.7399999999999998e-07 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438890+0000", 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.8889999999999999e-06 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.523 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438903+0000", 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2466000000000001e-05 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.438909+0000", 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.3640000000000004e-06 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439564+0000", 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000655084 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.524 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439571+0000", 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.472e-06 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448110+0000", 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0085393550000000002 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448181+0000", 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.1259000000000003e-05 2024-09-16T04:09:13.525 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.448725+0000", 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00054394000000000003 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.457803+0000", 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0090773959999999997 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.457955+0000", 2024-09-16T04:09:13.526 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00015255800000000001 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458049+0000", 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3930000000000004e-05 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "osd_op(client.4440.0:735 3.fs0 3:f7af954f:::smithi01639990-10:head [append 0~0] snapc 1a=[1a,18,16,15] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 21.001283579999999, 2024-09-16T04:09:13.527 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.019107652999999999, 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "client_info": { 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "client": "client.4440", 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "tid": 735 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:09:13.528 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439586+0000", 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439587+0000", 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.529 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439591+0000", 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.2570000000000001e-06 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439607+0000", 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.5846e-05 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.530 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439614+0000", 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.6579999999999996e-06 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439657+0000", 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 4.3402999999999998e-05 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.439663+0000", 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0959999999999997e-06 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.531 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449536+0000", 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0098724250000000006 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "started", 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449600+0000", 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4857999999999995e-05 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.449969+0000", 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00036863400000000001 2024-09-16T04:09:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458052+0000", 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.0080833569999999993 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "commit_sent", 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458113+0000", 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.0764999999999998e-05 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.533 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:52.458694+0000", 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.00058057699999999998 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 81/27 ECSubWriteReply(tid=222, last_complete=81'16, committed=1, applied=1))", 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.942376045, 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.59824786600000002, 2024-09-16T04:09:13.534 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.535 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498494+0000", 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 7.9100000000000003e-07 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:56.498500+0000", 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.3290000000000001e-06 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096610+0000", 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.59810986399999999 2024-09-16T04:09:13.536 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096617+0000", 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.9609999999999999e-06 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096710+0000", 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3670000000000003e-05 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.096741+0000", 2024-09-16T04:09:13.537 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.1251000000000003e-05 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 81/56 ECSubWriteReply(tid=223, last_complete=81'36, committed=1, applied=1))", 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:08:57.162644+0000", 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 16.278225396, 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.32306014599999999, 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.538 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162644+0000", 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162644+0000", 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.539 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162645+0000", 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.3699999999999999e-07 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.162658+0000", 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.3112000000000001e-05 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.540 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485612+0000", 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.32295339699999998 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485618+0000", 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 6.4489999999999996e-06 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485676+0000", 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 5.7414999999999998e-05 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.541 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:08:57.485704+0000", 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.8836e-05 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: "description": "MOSDECSubOpWrite(3.12s1 93/56 ECSubWrite(tid=239, reqid=osd.0.0:239, at_version=93'127, trim_to=0'0, roll_forward_to=92'125))", 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: "initiated_at": "2024-09-16T04:09:08.443504+0000", 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: "age": 4.9973653420000002, 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012380069, 2024-09-16T04:09:13.542 INFO:teuthology.orchestra.run.smithi174.stdout: "continuous": false, 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: "type_data": { 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: "events": [ 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "initiated", 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.443504+0000", 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "header_read", 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.443504+0000", 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0 2024-09-16T04:09:13.543 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "throttled", 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.443505+0000", 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.36e-07 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "all_read", 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.443526+0000", 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 2.1563999999999999e-05 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "dispatched", 2024-09-16T04:09:13.544 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.443563+0000", 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.7066000000000001e-05 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.443572+0000", 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 9.2019999999999993e-06 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "reached_pg", 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.443738+0000", 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.000165718 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.545 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.443750+0000", 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 1.2106999999999999e-05 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.455854+0000", 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 0.012103849999999999 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: "event": "done", 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: "time": "2024-09-16T04:09:08.455884+0000", 2024-09-16T04:09:13.546 INFO:teuthology.orchestra.run.smithi174.stdout: "duration": 3.0326000000000001e-05 2024-09-16T04:09:13.547 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.547 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.547 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.547 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-16T04:09:13.547 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-16T04:09:13.547 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-16T04:09:13.548 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T04:09:13.667 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:13.668 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:09:13.668 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:13.677 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T04:09:13.785 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:13.786 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:13.786 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:09:13.786 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:09:13.786 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:13.795 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T04:09:13.899 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:13.899 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:09:13.899 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:09:13.899 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:09:13.899 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.899 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 33/27 ECSubWrite(tid=86, reqid=client.4440.0:217, at_version=33'16, trim_to=0'0, roll_forward_to=30'12))", 2024-09-16T04:09:13.899 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 64.754907326999998, 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27979685999999998, 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.900 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141693+0000", 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141694+0000", 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5300000000000003e-07 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.141707+0000", 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3189e-05 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.901 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420849+0000", 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27914205600000003 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420859+0000", 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0156000000000001e-05 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420942+0000", 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2812000000000006e-05 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.902 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.420951+0000", 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0429999999999996e-06 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421458+0000", 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00050690700000000002 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.421490+0000", 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1943999999999998e-05 2024-09-16T04:09:13.903 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=163, reqid=client.4440.0:348, at_version=0'0, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 56.436851719000003, 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.126553044, 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.904 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459749+0000", 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459750+0000", 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6099999999999999e-07 2024-09-16T04:09:13.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.459763+0000", 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2695e-05 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585362+0000", 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.12559974199999999 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585388+0000", 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5758e-05 2024-09-16T04:09:13.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585504+0000", 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116153 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.585515+0000", 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0681e-05 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586270+0000", 2024-09-16T04:09:13.907 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00075517499999999997 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.586302+0000", 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1878999999999998e-05 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.908 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/27 ECSubWrite(tid=168, reqid=client.4440.0:356, at_version=42'38, trim_to=0'0, roll_forward_to=42'33))", 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 56.306309560000003, 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83830757199999995, 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.909 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590291+0000", 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590292+0000", 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0920000000000001e-06 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:17.590323+0000", 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1046000000000001e-05 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.910 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427874+0000", 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83755098900000002 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427883+0000", 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2040000000000003e-06 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427952+0000", 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8387999999999996e-05 2024-09-16T04:09:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.427961+0000", 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1870000000000003e-06 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428558+0000", 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000596987 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.428599+0000", 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0679000000000001e-05 2024-09-16T04:09:13.912 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:4(6),reservation_nonce:0)", 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.225387007000002, 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.158343295, 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671214+0000", 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5000000000000002e-07 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.914 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.671217+0000", 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3709999999999998e-06 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829436+0000", 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.15821966900000001 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829443+0000", 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0910000000000004e-06 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.915 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829496+0000", 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2938e-05 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829512+0000", 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.829557+0000", 2024-09-16T04:09:13.916 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5207000000000003e-05 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.9s0,GRANT,e:47,from:11(1),reservation_nonce:0)", 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.064982948999997, 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37719259500000002, 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.917 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.918 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831618+0000", 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.831622+0000", 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1200000000000002e-06 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208515+0000", 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.37689345299999999 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.919 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208522+0000", 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2740000000000002e-06 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208576+0000", 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4153999999999997e-05 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208594+0000", 2024-09-16T04:09:13.920 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7660999999999999e-05 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.208810+0000", 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000216209 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:429 3.9s0 3:92dcccbe:::smithi01639990-16:head [append 0~0] snapc d=[d,c,b] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 46.388878112, 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016927891, 2024-09-16T04:09:13.921 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.922 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.922 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.922 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:13.922 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:13.922 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.922 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 429 2024-09-16T04:09:13.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.922 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.923 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.923 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:13.923 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.923 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507723+0000", 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5899999999999996e-07 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507726+0000", 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7020000000000001e-06 2024-09-16T04:09:13.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507733+0000", 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2919999999999998e-06 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.507739+0000", 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4589999999999997e-06 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516622+0000", 2024-09-16T04:09:13.925 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0088837169999999993 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.516697+0000", 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5240000000000005e-05 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517131+0000", 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00043342000000000001 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.926 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.518161+0000", 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001030053 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524611+0000", 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0064503529999999998 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.524650+0000", 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9096e-05 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:463 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 19] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.367165086999997, 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020235912000000002, 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 463 2024-09-16T04:09:13.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529436+0000", 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.929 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529437+0000", 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8819999999999999e-06 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529445+0000", 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6680000000000001e-06 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529461+0000", 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5661000000000001e-05 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.930 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529469+0000", 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0080000000000002e-06 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529535+0000", 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6458999999999995e-05 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529666+0000", 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00013069399999999999 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.931 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.543531+0000", 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013865109 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549189+0000", 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056579109999999998 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549610+0000", 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00042142699999999999 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.932 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.549671+0000", 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1093000000000006e-05 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:464 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 45.367112859999999, 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016474697, 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.933 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 464 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.934 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529488+0000", 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3e-07 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529492+0000", 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.1259999999999998e-06 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.935 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529503+0000", 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1043000000000001e-05 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529509+0000", 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1550000000000004e-06 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529887+0000", 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00037831900000000001 2024-09-16T04:09:13.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.529937+0000", 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9976999999999999e-05 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545726+0000", 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01578887 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545842+0000", 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000115748 2024-09-16T04:09:13.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.545962+0000", 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000120829 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.11s6 68/56 ECSubWrite(tid=17, reqid=osd.10.0:17, at_version=68'19, trim_to=0'0, roll_forward_to=64'17))", 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.357426938, 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014242092, 2024-09-16T04:09:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539174+0000", 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8500000000000001e-07 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539199+0000", 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4397999999999999e-05 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539248+0000", 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9033000000000003e-05 2024-09-16T04:09:13.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539258+0000", 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8800000000000003e-06 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539858+0000", 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060046199999999996 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.539867+0000", 2024-09-16T04:09:13.941 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.363e-06 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553388+0000", 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013520249 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.553416+0000", 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8121999999999999e-05 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.942 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=64, reqid=osd.4.0:64, at_version=68'43, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.356312934000002, 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014089379000000001, 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.943 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540288+0000", 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1500000000000001e-07 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540308+0000", 2024-09-16T04:09:13.944 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0585999999999999e-05 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540337+0000", 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8560000000000001e-05 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540345+0000", 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6299999999999998e-06 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.945 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540400+0000", 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4818000000000003e-05 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.540411+0000", 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1009000000000001e-05 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554349+0000", 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013938394 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.946 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.554377+0000", 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8167e-05 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.14s1 68/48 ECSubWrite(tid=66, reqid=osd.4.0:66, at_version=68'45, trim_to=0'0, roll_forward_to=63'41))", 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.354471643, 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01391278, 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.947 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542129+0000", 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542130+0000", 2024-09-16T04:09:13.948 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6600000000000005e-07 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542154+0000", 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3862999999999998e-05 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542183+0000", 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9108999999999999e-05 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.949 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.542192+0000", 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4460000000000007e-06 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549962+0000", 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007769977 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.549972+0000", 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0074e-05 2024-09-16T04:09:13.950 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556017+0000", 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060444319999999998 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.556042+0000", 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5012999999999999e-05 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.951 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=119, reqid=client.4440.0:705, at_version=73'60, trim_to=0'0, roll_forward_to=73'57))", 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.401915568, 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035584957, 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.952 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494685+0000", 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494686+0000", 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.9999999999999996e-07 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494705+0000", 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9208999999999999e-05 2024-09-16T04:09:13.953 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529770+0000", 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035064924999999997 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529777+0000", 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5069999999999997e-06 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529831+0000", 2024-09-16T04:09:13.954 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4122000000000001e-05 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529845+0000", 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4571e-05 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530243+0000", 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00039740100000000003 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530270+0000", 2024-09-16T04:09:13.955 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7421999999999999e-05 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:708 3.15s0 3:ada5d29c:::smithi01639990-1:head [copy-from ver 69] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.384846027999998, 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49730772000000001, 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.956 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 708 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.957 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511755+0000", 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511756+0000", 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6029999999999999e-06 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511762+0000", 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2500000000000003e-06 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.958 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511774+0000", 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1761e-05 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511797+0000", 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2416999999999999e-05 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511872+0000", 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5437000000000003e-05 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.959 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511975+0000", 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010341899999999999 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.526578+0000", 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014602084 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.529422+0000", 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028444880000000001 2024-09-16T04:09:13.960 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008984+0000", 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47956190999999998 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009062+0000", 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8350999999999996e-05 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.961 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:709 3.15s0 3:ada5d29c:::smithi01639990-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.384790526, 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017140263999999999, 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 709 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.962 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511810+0000", 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0800000000000001e-07 2024-09-16T04:09:13.963 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511812+0000", 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7799999999999999e-06 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511816+0000", 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.172e-06 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.964 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.511819+0000", 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9739999999999998e-06 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512187+0000", 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00036796799999999998 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.512240+0000", 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3013999999999998e-05 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.965 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528792+0000", 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016551461999999999 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528863+0000", 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0797000000000007e-05 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.528950+0000", 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7788999999999996e-05 2024-09-16T04:09:13.966 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s7 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.365974009999999, 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.36295003300000001, 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.967 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530627+0000", 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7899999999999998e-07 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.968 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530637+0000", 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9180000000000006e-06 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892881+0000", 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.362243749 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.969 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892890+0000", 2024-09-16T04:09:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5620000000000004e-06 2024-09-16T04:09:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892950+0000", 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0270999999999999e-05 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.892959+0000", 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1689999999999998e-06 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893539+0000", 2024-09-16T04:09:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000579628 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.893577+0000", 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8056999999999998e-05 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=131, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 25.365689473, 2024-09-16T04:09:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47861527599999998, 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530911+0000", 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.975 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530912+0000", 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4400000000000001e-07 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530915+0000", 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5530000000000001e-06 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008849+0000", 2024-09-16T04:09:13.976 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.47793379000000002 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008854+0000", 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0340000000000004e-06 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.008927+0000", 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3088000000000002e-05 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.977 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.009526+0000", 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059936699999999996 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.15s0 73/48 ECSubWriteReply(tid=132, last_complete=73'61, committed=1, applied=1))", 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 24.886270202999999, 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44531548500000001, 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.978 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010330+0000", 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010331+0000", 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9600000000000001e-07 2024-09-16T04:09:13.979 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.010332+0000", 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.454e-06 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455554+0000", 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.44522236199999998 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.980 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455561+0000", 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4189999999999999e-06 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455608+0000", 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.6928000000000002e-05 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:49.455646+0000", 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8126000000000001e-05 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.981 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:819 3.9s0 3:94de6fe1:::smithi01639990-3:head [stat,delete] snapc 1e=[1e,1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:58.510155+0000", 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.386445533, 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.016397412, 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 819 2024-09-16T04:09:13.982 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510155+0000", 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510155+0000", 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.983 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510157+0000", 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.823e-06 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510167+0000", 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0237e-05 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510184+0000", 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7007000000000002e-05 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.984 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510193+0000", 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5050000000000007e-06 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510261+0000", 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8507e-05 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510270+0000", 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8890000000000008e-06 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.985 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.516074+0000", 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0058035379999999996 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.516179+0000", 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010501 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.518371+0000", 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021920770000000002 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.986 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.524696+0000", 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063251929999999998 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.526496+0000", 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001800486 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.526553+0000", 2024-09-16T04:09:13.987 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6140000000000001e-05 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 92/56 ECSubWriteReply(tid=194, last_complete=92'115, committed=1, applied=1))", 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.490709+0000", 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.4058912210000001, 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59931532099999996, 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.988 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.490709+0000", 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.490709+0000", 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.490710+0000", 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4099999999999998e-07 2024-09-16T04:09:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.490714+0000", 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5319999999999998e-06 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.089874+0000", 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59916044499999999 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.089888+0000", 2024-09-16T04:09:13.990 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3696999999999999e-05 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.089976+0000", 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8580999999999998e-05 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.090025+0000", 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8325000000000002e-05 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.991 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 92/56 ECSubWriteReply(tid=194, last_complete=92'115, committed=1, applied=1))", 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.494594+0000", 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.4020067620000001, 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.38089667199999999, 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.494594+0000", 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.992 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.494594+0000", 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.494595+0000", 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7599999999999996e-07 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.494600+0000", 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9899999999999997e-06 2024-09-16T04:09:13.993 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.875394+0000", 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.38079475000000002 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.875401+0000", 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2820000000000002e-06 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.875474+0000", 2024-09-16T04:09:13.994 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3679999999999999e-05 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.875491+0000", 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6093999999999999e-05 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:13.995 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:13.997 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T04:09:14.099 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:14.099 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:14.100 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:09:14.100 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:14.108 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T04:09:14.210 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:09:14.210 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:09:14.210 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:14.219 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T04:09:14.323 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:14.323 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 32/27 ECSubWrite(tid=14, reqid=client.4440.0:181, at_version=32'6, trim_to=0'0, roll_forward_to=30'4))", 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 67.226824156000006, 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.592222048, 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.324 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094307+0000", 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094308+0000", 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7e-07 2024-09-16T04:09:14.325 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.094326+0000", 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8389e-05 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685846+0000", 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.59152032600000004 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685854+0000", 2024-09-16T04:09:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.875e-06 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685904+0000", 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9258999999999997e-05 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.685911+0000", 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9600000000000003e-06 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.327 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686501+0000", 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00059033000000000002 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:07.686529+0000", 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8538999999999999e-05 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.2s1 42/27 ECSubRead(tid=170, to_read={3:4e1a0922:::smithi01639990-46:head=0,4096,0}, subchunks={3:4e1a0922:::smithi01639990-46:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:09:14.328 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 55.891307185999999, 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45829662500000001, 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429824+0000", 2024-09-16T04:09:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429825+0000", 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3099999999999996e-07 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.429845+0000", 2024-09-16T04:09:14.330 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9358e-05 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887829+0000", 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.45798399299999998 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.887847+0000", 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8427000000000002e-05 2024-09-16T04:09:14.331 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888027+0000", 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017955299999999999 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.888121+0000", 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4462999999999993e-05 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.332 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/27 ECSubWrite(tid=173, reqid=osd.3.0:173, at_version=43'42, trim_to=0'0, roll_forward_to=42'38))", 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 55.431356958999999, 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24596948399999999, 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.333 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889775+0000", 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7700000000000004e-07 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:18.889796+0000", 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0956999999999999e-05 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.334 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134891+0000", 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.24509425800000001 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134902+0000", 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1041e-05 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134988+0000", 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.5946999999999996e-05 2024-09-16T04:09:14.335 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.134996+0000", 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0299999999999994e-06 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135715+0000", 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00071893800000000002 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:19.135744+0000", 2024-09-16T04:09:14.336 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9536e-05 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.0s1,QREQUEST,e:47,from:11(0),reservation_nonce:8)", 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.107844354000001, 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.232821636, 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.337 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213287+0000", 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213289+0000", 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.361e-06 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.338 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.213297+0000", 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.833e-06 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445990+0000", 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.23269227200000001 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.445996+0000", 2024-09-16T04:09:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8719999999999999e-06 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446063+0000", 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6957000000000003e-05 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446081+0000", 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8440999999999998e-05 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.340 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.446109+0000", 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7900000000000001e-05 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDPGPush(3.bs7 57/56 [PushOp(3:d683cb60:::smithi01639990-49:head, version: 53'38, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d683cb60:::smithi01639990-49:head@53'38, size: 2310144, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 41.760363857000002, 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.043702234999999999, 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.341 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560768+0000", 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.342 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.560769+0000", 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0049999999999999e-06 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.561301+0000", 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00053276500000000004 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603955+0000", 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.042653605999999997 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.343 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.603967+0000", 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2065000000000001e-05 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604063+0000", 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5599000000000004e-05 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.604470+0000", 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00040719500000000001 2024-09-16T04:09:14.344 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=252, reqid=osd.3.0:252, at_version=57'65, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 41.411637716999998, 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27497091299999998, 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.345 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909494+0000", 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8299999999999997e-07 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.346 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.909517+0000", 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2886000000000001e-05 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183847+0000", 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.27432981400000001 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183853+0000", 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2929999999999997e-06 2024-09-16T04:09:14.347 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183902+0000", 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9008999999999998e-05 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.183911+0000", 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0989999999999999e-06 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184435+0000", 2024-09-16T04:09:14.348 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052493699999999995 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.184465+0000", 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9292e-05 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s1 57/56 ECSubWrite(tid=253, reqid=osd.3.0:253, at_version=57'67, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 41.411009686, 2024-09-16T04:09:14.349 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50851210899999999, 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910122+0000", 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.350 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910123+0000", 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8299999999999996e-07 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:32.910136+0000", 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.331e-05 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416910+0000", 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50677369000000005 2024-09-16T04:09:14.351 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416919+0000", 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8290000000000003e-06 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416975+0000", 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5983999999999998e-05 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.416985+0000", 2024-09-16T04:09:14.352 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5089999999999999e-06 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418599+0000", 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001614515 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:33.418634+0000", 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4588999999999997e-05 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.353 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 64/56 ECSubWrite(tid=285, reqid=client.4440.0:535, at_version=0'0, trim_to=0'0, roll_forward_to=64'72))", 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 34.779562617000003, 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012095003, 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.354 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541569+0000", 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541570+0000", 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.173e-06 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.541582+0000", 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2106999999999999e-05 2024-09-16T04:09:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552877+0000", 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011294692 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552889+0000", 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1875000000000001e-05 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552958+0000", 2024-09-16T04:09:14.356 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9510000000000004e-05 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.552967+0000", 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6240000000000008e-06 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553631+0000", 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00066445299999999996 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.357 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:39.553664+0000", 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2568999999999999e-05 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.3s6 68/46 ECSubWrite(tid=65, reqid=osd.4.0:65, at_version=68'22, trim_to=0'0, roll_forward_to=67'20))", 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 30.779836359000001, 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010787494999999999, 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.358 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541295+0000", 2024-09-16T04:09:14.359 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8699999999999999e-07 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541313+0000", 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7431e-05 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541369+0000", 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6311999999999999e-05 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.541376+0000", 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5729999999999996e-06 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544486+0000", 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031104150000000001 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.544495+0000", 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0960000000000007e-06 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552010+0000", 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075148200000000002 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:43.552083+0000", 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2661000000000001e-05 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.bs7 69/56 ECSubWrite(tid=338, reqid=osd.3.0:338, at_version=69'72, trim_to=0'0, roll_forward_to=68'68))", 2024-09-16T04:09:14.362 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.776744410999999, 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010064081000000001, 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.363 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544387+0000", 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544388+0000", 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9800000000000003e-07 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544409+0000", 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0993e-05 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.364 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544442+0000", 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2937e-05 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.544449+0000", 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.588e-06 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547347+0000", 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0028976599999999998 2024-09-16T04:09:14.365 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.547357+0000", 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6479999999999998e-06 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554419+0000", 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070620819999999999 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.366 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.554451+0000", 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2475000000000003e-05 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.13s6 69/56 ECSubWrite(tid=202, reqid=osd.7.0:202, at_version=69'26, trim_to=0'0, roll_forward_to=68'24))", 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 29.775407615999999, 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010752746000000001, 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.367 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545724+0000", 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.368 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545725+0000", 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4199999999999995e-07 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.545750+0000", 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.4915000000000001e-05 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546880+0000", 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0011307870000000001 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.369 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.546891+0000", 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0825000000000001e-05 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551495+0000", 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004604131 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.551504+0000", 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6230000000000003e-06 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.370 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556450+0000", 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049460210000000001 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:44.556477+0000", 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6801999999999999e-05 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.371 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.6s6 82/56 ECSubWrite(tid=197, reqid=client.4440.0:785, at_version=82'40, trim_to=0'0, roll_forward_to=74'38))", 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.828364979, 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0095796900000000001, 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.372 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492767+0000", 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6400000000000001e-07 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492863+0000", 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6371000000000003e-05 2024-09-16T04:09:14.373 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492883+0000", 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9507000000000001e-05 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492894+0000", 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0998999999999999e-05 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492958+0000", 2024-09-16T04:09:14.374 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3898000000000004e-05 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.492968+0000", 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0012e-05 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.502307+0000", 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0093395509999999998 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.502346+0000", 2024-09-16T04:09:14.375 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9187999999999998e-05 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.17s2 83/56 ECSubWrite(tid=206, reqid=client.4440.0:818, at_version=0'0, trim_to=0'0, roll_forward_to=83'73))", 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:58.510113+0000", 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 15.811018526, 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.57355782399999999, 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.376 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510113+0000", 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510113+0000", 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510114+0000", 2024-09-16T04:09:14.377 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.68e-07 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.510124+0000", 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0117e-05 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082885+0000", 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.57276119999999997 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.378 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082895+0000", 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7580000000000004e-06 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082959+0000", 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4545e-05 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.082968+0000", 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9250000000000001e-06 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.379 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.083638+0000", 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00067024900000000004 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.083671+0000", 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.2462000000000002e-05 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.1s5 92/56 ECSubWrite(tid=226, reqid=osd.0.0:226, at_version=92'36, trim_to=0'0, roll_forward_to=89'34))", 2024-09-16T04:09:14.380 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.452733+0000", 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.8683990450000003, 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010158257, 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452733+0000", 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.381 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452733+0000", 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452733+0000", 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1299999999999999e-07 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452766+0000", 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3034000000000003e-05 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.382 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452853+0000", 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6799999999999996e-05 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452867+0000", 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4837e-05 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452932+0000", 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4726000000000003e-05 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.383 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.452942+0000", 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0132999999999999e-05 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.462867+0000", 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.009924782 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.462891+0000", 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3731999999999999e-05 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.384 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s5 92/56 ECSubWrite(tid=440, reqid=osd.3.0:440, at_version=92'95, trim_to=0'0, roll_forward_to=89'93))", 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.453317+0000", 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.867814847, 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011870421000000001, 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453317+0000", 2024-09-16T04:09:14.385 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453317+0000", 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453318+0000", 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1149999999999999e-06 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.386 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453343+0000", 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5352000000000001e-05 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453514+0000", 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00017035999999999999 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453523+0000", 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2350000000000005e-06 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453600+0000", 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7463e-05 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453611+0000", 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0648e-05 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.465151+0000", 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011540478999999999 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.388 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.465187+0000", 2024-09-16T04:09:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.5769000000000003e-05 2024-09-16T04:09:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.389 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.4s2 92/56 ECSubWrite(tid=53, reqid=osd.10.0:53, at_version=92'41, trim_to=0'0, roll_forward_to=89'39))", 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.453423+0000", 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.8677081019999999, 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0095119709999999993, 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453423+0000", 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.390 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453423+0000", 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453424+0000", 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0900000000000001e-07 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.453452+0000", 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8013000000000001e-05 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.391 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.454574+0000", 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00112243 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.454583+0000", 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8240000000000006e-06 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.454655+0000", 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1883000000000003e-05 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.392 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.454664+0000", 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9770000000000006e-06 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.462901+0000", 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082373819999999997 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.462935+0000", 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4252999999999999e-05 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.393 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.15s7 92/48 ECSubWrite(tid=190, reqid=osd.8.0:190, at_version=92'83, trim_to=0'0, roll_forward_to=89'81))", 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.459701+0000", 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.8614306999999997, 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.026074818, 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.394 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.459701+0000", 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.459701+0000", 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.459701+0000", 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8999999999999996e-07 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.395 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.459720+0000", 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8476999999999999e-05 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.478473+0000", 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.018753216999999999 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.396 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.478483+0000", 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6169999999999996e-06 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.478577+0000", 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4059000000000002e-05 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.478589+0000", 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2663999999999999e-05 2024-09-16T04:09:14.397 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.485750+0000", 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007160596 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.485776+0000", 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5598000000000001e-05 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.398 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.9s3 92/56 ECSubWrite(tid=192, reqid=osd.8.0:192, at_version=92'115, trim_to=0'0, roll_forward_to=92'113))", 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:07.479459+0000", 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 6.8416725950000004, 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0095768099999999998, 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.479459+0000", 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.399 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.479459+0000", 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.479460+0000", 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9100000000000003e-07 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.479476+0000", 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6595999999999998e-05 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.400 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.486477+0000", 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0070005110000000001 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.486488+0000", 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1017e-05 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.486549+0000", 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1069999999999996e-05 2024-09-16T04:09:14.401 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.486560+0000", 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0674e-05 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.488988+0000", 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0024284210000000001 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:07.489036+0000", 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7830000000000001e-05 2024-09-16T04:09:14.402 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as5 93/56 ECSubWrite(tid=81, reqid=osd.6.0:81, at_version=93'67, trim_to=0'0, roll_forward_to=92'65))", 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:08.457204+0000", 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.863927747, 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.036487762, 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.403 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.457204+0000", 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.457204+0000", 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.457204+0000", 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2699999999999999e-07 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.404 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.457228+0000", 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3234000000000002e-05 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.487257+0000", 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.03002954 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.487266+0000", 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9430000000000006e-06 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.405 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.487335+0000", 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9196000000000006e-05 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.487350+0000", 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4561e-05 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.493666+0000", 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0063165119999999998 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.406 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.493692+0000", 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5148999999999999e-05 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.as5 93/56 ECSubWrite(tid=83, reqid=client.4440.0:930, at_version=0'0, trim_to=0'0, roll_forward_to=93'69))", 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:08.496719+0000", 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 5.8244125740000001, 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.060932394000000001, 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.496719+0000", 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.496719+0000", 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.408 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.496720+0000", 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.92e-07 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.496730+0000", 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0077e-05 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.556875+0000", 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.060145123000000002 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.556885+0000", 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8980000000000008e-06 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.556947+0000", 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2687000000000005e-05 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.556957+0000", 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9899999999999992e-06 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.410 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.557615+0000", 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00065753600000000001 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:08.557651+0000", 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6390999999999998e-05 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.411 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:14.413 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T04:09:14.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T04:09:14.516 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:14.516 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:14.516 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:09:14.516 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:14.524 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T04:09:14.630 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:14.630 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:14.630 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:09:14.630 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:09:14.630 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:14.639 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T04:09:14.743 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:14.743 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:09:14.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:09:14.743 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:09:14.743 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.743 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.7s4 52/46 ECSubWrite(tid=210, reqid=client.4440.0:434, at_version=52'63, trim_to=0'0, roll_forward_to=50'60))", 2024-09-16T04:09:14.743 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:09:14.743 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 47.222623853999998, 2024-09-16T04:09:14.743 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83621405000000004, 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:09:14.744 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517767+0000", 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.8200000000000001e-07 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:27.517790+0000", 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2540000000000001e-05 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.745 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352789+0000", 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.83499984500000002 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352804+0000", 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4569999999999999e-05 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352928+0000", 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00012396799999999999 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.746 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.352943+0000", 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5369000000000001e-05 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353943+0000", 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099977500000000006 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.353981+0000", 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7700999999999997e-05 2024-09-16T04:09:14.747 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.2s7 53/46 ECSubWrite(tid=220, reqid=client.4440.0:452, at_version=53'61, trim_to=0'0, roll_forward_to=50'56))", 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 46.220620300999997, 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51827822899999998, 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.748 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519770+0000", 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519771+0000", 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.61e-07 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.749 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.519788+0000", 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6699999999999999e-05 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036882+0000", 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51709414300000001 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036895+0000", 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3363999999999999e-05 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.750 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036964+0000", 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.8734999999999998e-05 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.036973+0000", 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0289999999999994e-06 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038014+0000", 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010410529999999999 2024-09-16T04:09:14.751 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.038049+0000", 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4443999999999998e-05 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:526 3.4s0 3:26eb5942:::smithi01639990-23:head [append 0~622592] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 37.109823913, 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01502261, 2024-09-16T04:09:14.752 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 526 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.753 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630567+0000", 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.630568+0000", 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5680000000000001e-06 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631353+0000", 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00078490700000000005 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.754 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631367+0000", 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3441000000000001e-05 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631375+0000", 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8809999999999996e-06 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631465+0000", 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0673000000000004e-05 2024-09-16T04:09:14.755 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631565+0000", 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9753000000000005e-05 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.632641+0000", 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001076097 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634418+0000", 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017769680000000001 2024-09-16T04:09:14.756 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645537+0000", 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011119393999999999 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.645589+0000", 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1928000000000002e-05 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.757 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:527 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=7b] snapc 12=[12,11,10] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 37.10898684, 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015053211, 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 527 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.758 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631404+0000", 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631405+0000", 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.08e-06 2024-09-16T04:09:14.759 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631411+0000", 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9009999999999999e-06 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631419+0000", 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4090000000000003e-06 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631426+0000", 2024-09-16T04:09:14.760 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4999999999999996e-06 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633558+0000", 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002131796 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.633663+0000", 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000105833 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634106+0000", 2024-09-16T04:09:14.761 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00044296899999999998 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.636380+0000", 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022740849999999999 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646383+0000", 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.01000238 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.762 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646457+0000", 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4258000000000006e-05 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:528 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 37.108912259, 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.65958352099999995, 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 528 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631478+0000", 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.764 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631479+0000", 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9599999999999998e-07 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631483+0000", 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8160000000000004e-06 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631494+0000", 2024-09-16T04:09:14.765 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0407999999999999e-05 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.631501+0000", 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2269999999999996e-06 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634655+0000", 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031544339999999998 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.634732+0000", 2024-09-16T04:09:14.766 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6495999999999994e-05 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646864+0000", 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012131807999999999 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.646965+0000", 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010173 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.291062+0000", 2024-09-16T04:09:14.767 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64409670600000002 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:695 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~589824] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.291364266999999, 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014512838, 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.768 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 695 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449026+0000", 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.769 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.449028+0000", 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.917e-06 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450009+0000", 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000980367 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450023+0000", 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.416e-05 2024-09-16T04:09:14.770 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450032+0000", 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5820000000000008e-06 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450094+0000", 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1698000000000005e-05 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.450210+0000", 2024-09-16T04:09:14.771 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000116104 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.451973+0000", 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001762958 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463392+0000", 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011419024999999999 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.772 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463450+0000", 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8251999999999997e-05 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463539+0000", 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8775000000000003e-05 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:696 3.11s0 3:8bfd968c:::smithi01639990-27:head [append 0~753664] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:09:14.773 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.288231833000001, 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013764281, 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 696 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:09:14.774 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452159+0000", 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.452161+0000", 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2579999999999998e-06 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453157+0000", 2024-09-16T04:09:14.775 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00099543299999999999 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453172+0000", 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5917e-05 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.453182+0000", 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3959999999999995e-06 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.776 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455292+0000", 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0021103379999999998 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.455406+0000", 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000114203 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.456783+0000", 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001376377 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.777 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.463544+0000", 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067608640000000001 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465825+0000", 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0022816350000000002 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.465923+0000", 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7860000000000002e-05 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.778 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:700 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.269081324999998, 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.014504263, 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 700 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.779 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471309+0000", 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471310+0000", 2024-09-16T04:09:14.780 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.08e-07 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471312+0000", 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.0540000000000002e-06 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471318+0000", 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5080000000000002e-06 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.471322+0000", 2024-09-16T04:09:14.781 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.3629999999999998e-06 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472045+0000", 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00072327100000000005 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.472133+0000", 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8528000000000003e-05 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.782 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484130+0000", 2024-09-16T04:09:14.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.011996163000000001 2024-09-16T04:09:14.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.783 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.783 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.484213+0000", 2024-09-16T04:09:14.783 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3800000000000004e-05 2024-09-16T04:09:14.783 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.783 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.784 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.784 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.485814+0000", 2024-09-16T04:09:14.784 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0016001679999999999 2024-09-16T04:09:14.784 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.784 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.784 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.784 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.784 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWrite(3.8s6 73/56 ECSubWrite(tid=120, reqid=client.4440.0:705, at_version=0'0, trim_to=0'0, roll_forward_to=73'60))", 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.209756429999999, 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095992542, 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.785 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530634+0000", 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530636+0000", 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.522e-06 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530651+0000", 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4980999999999999e-05 2024-09-16T04:09:14.786 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625860+0000", 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.095209590999999996 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625873+0000", 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2413e-05 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625978+0000", 2024-09-16T04:09:14.787 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010492099999999999 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.625994+0000", 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6058000000000001e-05 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626583+0000", 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00058931700000000001 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.788 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.626627+0000", 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3739000000000003e-05 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:803 3.4s0 3:26eb5942:::smithi01639990-23:head [setxattr _header (57) in=64b] snapc 1d=[1d,1c,1b,16,15] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.178218251000001, 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.77068146599999998, 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:14.789 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 803 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562172+0000", 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.790 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562173+0000", 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3200000000000005e-07 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562176+0000", 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9900000000000002e-06 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562183+0000", 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9480000000000002e-06 2024-09-16T04:09:14.791 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562188+0000", 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1479999999999997e-06 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.565211+0000", 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0030228519999999999 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.565509+0000", 2024-09-16T04:09:14.792 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000297578 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.566353+0000", 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00084484499999999999 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.568092+0000", 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0017385720000000001 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:14.793 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.332782+0000", 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.76468974099999998 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.332854+0000", 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.216e-05 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:804 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.178194699999999, 2024-09-16T04:09:14.794 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4794380249999999, 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 804 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.795 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562196+0000", 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2e-07 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562199+0000", 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1119999999999999e-06 2024-09-16T04:09:14.796 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562206+0000", 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9369999999999998e-06 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.562211+0000", 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2569999999999998e-06 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.567444+0000", 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.005233265 2024-09-16T04:09:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.567504+0000", 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9769999999999999e-05 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.333251+0000", 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.76574722100000003 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.333355+0000", 2024-09-16T04:09:14.798 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010408600000000001 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041634+0000", 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.70827885700000004 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 82/56 ECSubWriteReply(tid=41, last_complete=82'37, committed=1, applied=1))", 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:57.572277+0000", 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 17.168113793, 2024-09-16T04:09:14.799 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.754736132, 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572277+0000", 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572277+0000", 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.800 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572278+0000", 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5199999999999996e-07 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:57.572281+0000", 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.7290000000000002e-06 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.326881+0000", 2024-09-16T04:09:14.801 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.75459961200000003 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.326897+0000", 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6180000000000001e-05 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.326988+0000", 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0477999999999997e-05 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.327013+0000", 2024-09-16T04:09:14.802 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.5480999999999999e-05 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpReadReply(3.4s0 82/56 ECSubReadReply(tid=43, attrs_read=0))", 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:58.334354+0000", 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 16.406036919000002, 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.70728895800000002, 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.803 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.334354+0000", 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.334354+0000", 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.334354+0000", 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.06e-07 2024-09-16T04:09:14.804 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:58.334369+0000", 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5041e-05 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041321+0000", 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.70695176299999996 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041348+0000", 2024-09-16T04:09:14.805 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6659999999999999e-05 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041459+0000", 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011134200000000001 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:59.041643+0000", 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00018354600000000001 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.806 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:848 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~4063232,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.513588+0000", 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.226802340000001, 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.029163938, 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 848 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.807 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513588+0000", 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513588+0000", 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513590+0000", 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.8470000000000001e-06 2024-09-16T04:09:14.808 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513600+0000", 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3619999999999995e-06 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513616+0000", 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6758000000000001e-05 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513633+0000", 2024-09-16T04:09:14.809 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7139999999999999e-05 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513708+0000", 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4762e-05 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513717+0000", 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4870000000000002e-06 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520258+0000", 2024-09-16T04:09:14.810 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0065412760000000004 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520352+0000", 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3752000000000006e-05 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.542752+0000", 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.022400554 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.811 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:849 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~4063232,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.513662+0000", 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.22672888, 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.032938179999999997, 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 849 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513662+0000", 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513662+0000", 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513663+0000", 2024-09-16T04:09:14.813 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1999999999999999e-07 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513668+0000", 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.835e-06 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513677+0000", 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6810000000000004e-06 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.814 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513685+0000", 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3280000000000006e-06 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514004+0000", 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000318146 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514011+0000", 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3599999999999998e-06 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.815 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520894+0000", 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0068835809999999997 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520991+0000", 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.6646999999999998e-05 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.546600+0000", 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025608881999999999 2024-09-16T04:09:14.816 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:850 3.4s0 3:26eb5942:::smithi01639990-23:head [read 0~4063232,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.513985+0000", 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.22640541, 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035758587000000001, 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.817 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 850 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513985+0000", 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.818 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513985+0000", 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513986+0000", 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1739999999999999e-06 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514004+0000", 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.7393000000000001e-05 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.819 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514014+0000", 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8859999999999999e-06 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514021+0000", 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.6580000000000002e-06 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514084+0000", 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2485999999999998e-05 2024-09-16T04:09:14.820 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514105+0000", 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1185000000000001e-05 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.524619+0000", 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.010513504999999999 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.524728+0000", 2024-09-16T04:09:14.821 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000109324 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.549744+0000", 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025015975999999999 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:851 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~2146304,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.514064+0000", 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.226327040999999, 2024-09-16T04:09:14.822 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.013147891, 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 851 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514064+0000", 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.823 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514064+0000", 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514065+0000", 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.569999999999999e-07 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514070+0000", 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8320000000000002e-06 2024-09-16T04:09:14.824 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514078+0000", 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.3939999999999998e-06 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514083+0000", 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.8060000000000002e-06 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514154+0000", 2024-09-16T04:09:14.825 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1019000000000006e-05 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514169+0000", 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.5112e-05 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.519800+0000", 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0056317049999999999 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.826 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.519945+0000", 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000144155 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.527212+0000", 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.007266911 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:852 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.514096+0000", 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.226295155000001, 2024-09-16T04:09:14.827 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015213175000000001, 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 852 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514096+0000", 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.828 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514096+0000", 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514096+0000", 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514109+0000", 2024-09-16T04:09:14.829 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2979999999999999e-05 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514118+0000", 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.8710000000000003e-06 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514124+0000", 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2700000000000001e-06 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.830 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514211+0000", 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6284000000000003e-05 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514217+0000", 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0499999999999997e-06 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520494+0000", 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0062772829999999998 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.831 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520602+0000", 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010760999999999999 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.529309+0000", 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0087071709999999997 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.832 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:853 3.11s0 3:8bfd968c:::smithi01639990-27:head [read 0~2146304,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.514138+0000", 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.226252904000001, 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017460188000000001, 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 853 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.833 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514138+0000", 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514138+0000", 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514138+0000", 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3900000000000004e-07 2024-09-16T04:09:14.834 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514143+0000", 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3409999999999996e-06 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514151+0000", 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0290000000000005e-06 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514157+0000", 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.3990000000000001e-06 2024-09-16T04:09:14.835 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514251+0000", 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3330000000000003e-05 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514256+0000", 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4240000000000001e-06 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.522521+0000", 2024-09-16T04:09:14.836 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0082649720000000006 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.522639+0000", 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000118114 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.531598+0000", 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0089589400000000003 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.837 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpReadReply(3.4s0 86/56 ECSubReadReply(tid=46, attrs_read=0))", 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.526934+0000", 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.213456851, 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.015870366, 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.526934+0000", 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.838 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.526934+0000", 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.526935+0000", 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2399999999999997e-07 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.536580+0000", 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0096454929999999998 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.839 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.536677+0000", 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7235e-05 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.536687+0000", 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8579999999999995e-06 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.536843+0000", 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00015625799999999999 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:14.840 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:14.841 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:14.841 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.542804+0000", 2024-09-16T04:09:14.841 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0059607979999999998 2024-09-16T04:09:14.841 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.841 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.841 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.841 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:14.841 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:14.841 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:14.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T04:09:14.946 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:14.946 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:14.946 INFO:teuthology.orchestra.run.smithi194.stdout: "num_ops": 0 2024-09-16T04:09:14.946 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:14.954 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T04:09:15.056 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:15.056 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [], 2024-09-16T04:09:15.056 INFO:teuthology.orchestra.run.smithi194.stdout: "complaint_time": 30, 2024-09-16T04:09:15.057 INFO:teuthology.orchestra.run.smithi194.stdout: "num_blocked_ops": 0 2024-09-16T04:09:15.057 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:15.065 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T04:09:15.169 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-16T04:09:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "size": 20, 2024-09-16T04:09:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 600, 2024-09-16T04:09:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "ops": [ 2024-09-16T04:09:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_repop(client.4192.0:124 1.0 e22/19 1:30306672:devicehealth::main.db.0000000000000000:head v 22'86, mlcod=21'84)", 2024-09-16T04:09:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 82.735588747999998, 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.020095396000000001, 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430781+0000", 2024-09-16T04:09:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430782+0000", 2024-09-16T04:09:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9699999999999996e-07 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.430789+0000", 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4660000000000002e-06 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450028+0000", 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019238816999999998 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450042+0000", 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.4487e-05 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450169+0000", 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000126787 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450188+0000", 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9145000000000001e-05 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450816+0000", 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00062741399999999997 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:07:52.450877+0000", 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0782999999999997e-05 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:197 3.es0 3:7f241f48:::smithi01639990-30:head [copy-from ver 23] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 66.956379400000003, 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.023242381999999999, 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 197 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209991+0000", 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6499999999999999e-07 2024-09-16T04:09:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.209995+0000", 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.388e-06 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210002+0000", 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5209999999999997e-06 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210008+0000", 2024-09-16T04:09:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.8780000000000003e-06 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210046+0000", 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.8421999999999999e-05 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210099+0000", 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2469999999999997e-05 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.227450+0000", 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.017351235 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.232352+0000", 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0049017239999999997 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233168+0000", 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00081616600000000001 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.233233+0000", 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5012999999999996e-05 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:198 3.es0 3:7f241f48:::smithi01639990-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 66.956349091000007, 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019673758, 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 198 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210021+0000", 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.3099999999999998e-07 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210024+0000", 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2620000000000001e-06 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210030+0000", 2024-09-16T04:09:15.182 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9020000000000002e-06 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210037+0000", 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.1480000000000003e-06 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210227+0000", 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00019055499999999999 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for blocked object", 2024-09-16T04:09:15.183 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.210269+0000", 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2246999999999997e-05 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229480+0000", 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.019210896000000002 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229587+0000", 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107026 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:08.229695+0000", 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000107291 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:213 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (56) in=7b] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 66.050270585999996, 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53275775299999995, 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:15.185 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 213 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116099+0000", 2024-09-16T04:09:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116100+0000", 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.5600000000000005e-07 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116103+0000", 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.174e-06 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116109+0000", 2024-09-16T04:09:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.4330000000000003e-06 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116114+0000", 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1399999999999999e-06 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119298+0000", 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0031846729999999998 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119365+0000", 2024-09-16T04:09:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6484999999999998e-05 2024-09-16T04:09:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.119675+0000", 2024-09-16T04:09:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00031021999999999999 2024-09-16T04:09:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.122365+0000", 2024-09-16T04:09:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.002689851 2024-09-16T04:09:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648785+0000", 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52641971300000001 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.648857+0000", 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2408000000000002e-05 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:214 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e33)", 2024-09-16T04:09:15.193 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 66.050247909999996, 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.53416417900000002, 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 214 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:09:15.194 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116122+0000", 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.2100000000000002e-07 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.195 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116125+0000", 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2649999999999999e-06 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116132+0000", 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5190000000000003e-06 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.116137+0000", 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9869999999999999e-06 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.196 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120036+0000", 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0038984979999999998 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.120118+0000", 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2312999999999996e-05 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649191+0000", 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.52907320199999996 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.197 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.649279+0000", 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7585999999999998e-05 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:09.650286+0000", 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0010073879999999999 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "replica_scrub(pg: 3.4s1,from:0'0,to:37'17,epoch:47/46,start:3:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T04:09:15.198 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 52.308446373000002, 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16799329900000001, 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:09:15.199 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857924+0000", 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9400000000000005e-07 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:22.857929+0000", 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.4719999999999997e-06 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.200 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025793+0000", 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.16786378299999999 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025803+0000", 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.9629999999999993e-06 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025875+0000", 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.2914e-05 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.201 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025887+0000", 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1345999999999999e-05 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.025917+0000", 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.0127000000000001e-05 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDScrubReserve(3.4s1,RELEASE,e:47,from:1(0),reservation_nonce:0)", 2024-09-16T04:09:15.202 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 52.139301398000001, 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26135687200000002, 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.203 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027069+0000", 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.2800000000000005e-07 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.027075+0000", 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.3859999999999998e-06 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.204 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288302+0000", 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.26122710999999998 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288308+0000", 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6189999999999999e-06 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288392+0000", 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.4549000000000007e-05 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.205 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288414+0000", 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.1576000000000001e-05 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.288425+0000", 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.1704e-05 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:387 3.8s0 3:17b72cd6:::smithi01639990-47:head [append 0~0] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T04:09:15.206 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.700199292000001, 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.51761721800000005, 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 387 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.207 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466171+0000", 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6399999999999998e-07 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.208 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466173+0000", 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.396e-06 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466179+0000", 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9290000000000003e-06 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466184+0000", 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.9760000000000003e-06 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.209 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466397+0000", 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00021269400000000001 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.466406+0000", 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0769999999999997e-06 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479302+0000", 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.012896014000000001 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.210 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479415+0000", 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000112767 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.479914+0000", 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00049861199999999995 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484063+0000", 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.004149158 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.211 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983716+0000", 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49965376500000003 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983788+0000", 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.1465999999999996e-05 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.212 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/46 ECSubWriteReply(tid=59, last_complete=48'38, committed=1, applied=1))", 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.682158280000003, 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.50006588100000005, 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.213 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484212+0000", 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.2499999999999999e-07 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.484214+0000", 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.6440000000000001e-06 2024-09-16T04:09:15.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983573+0000", 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.49935939200000001 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983581+0000", 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5780000000000001e-06 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.983628+0000", 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.7642999999999998e-05 2024-09-16T04:09:15.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.984278+0000", 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00064939900000000005 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "rep_scrubmap(3.8s0 e46 from shard 10(6))", 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 51.180056987999997, 2024-09-16T04:09:15.216 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21666559099999999, 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986313+0000", 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.217 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986314+0000", 2024-09-16T04:09:15.218 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.5600000000000005e-07 2024-09-16T04:09:15.218 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.218 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.218 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.218 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:23.986319+0000", 2024-09-16T04:09:15.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.0950000000000003e-06 2024-09-16T04:09:15.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.219 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202862+0000", 2024-09-16T04:09:15.219 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.21654300900000001 2024-09-16T04:09:15.219 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.219 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.219 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202872+0000", 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0077e-05 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202931+0000", 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9568000000000003e-05 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202942+0000", 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0139999999999999e-05 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.220 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:24.202979+0000", 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.6946000000000002e-05 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:446 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc d=[d,b] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 46.662830540999998, 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67862900900000001, 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.221 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 446 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.222 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503539+0000", 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503541+0000", 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3680000000000001e-06 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503545+0000", 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.561e-06 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.223 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503553+0000", 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8939999999999993e-06 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503559+0000", 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.224e-06 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.224 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505239+0000", 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.001679739 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505303+0000", 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4219000000000003e-05 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505673+0000", 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000369374 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.225 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.507082+0000", 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014090330000000001 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182055+0000", 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67497357700000005 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182168+0000", 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00011302 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.226 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:447 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 46.662801035000001, 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.680085987, 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.227 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 447 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503569+0000", 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.228 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503570+0000", 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.44e-07 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503576+0000", 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.4559999999999997e-06 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503602+0000", 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6264000000000001e-05 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.229 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.503610+0000", 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.4109999999999999e-06 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.505945+0000", 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0023353390000000001 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.506004+0000", 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9410999999999998e-05 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.230 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182603+0000", 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67659811199999997 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182695+0000", 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.1951000000000001e-05 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.231 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.183655+0000", 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00096039900000000002 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 46.657728710000001, 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.299096274, 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.232 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508641+0000", 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.233 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508642+0000", 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5400000000000002e-07 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.508645+0000", 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.9249999999999999e-06 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807520+0000", 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.29887539899999999 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.234 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807534+0000", 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.3643e-05 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807681+0000", 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00014696699999999999 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.807738+0000", 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.6885999999999997e-05 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 53/46 ECSubWriteReply(tid=67, last_complete=53'47, committed=1, applied=1))", 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 46.657347244999997, 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67354323000000005, 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509023+0000", 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.9500000000000002e-07 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.237 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:28.509027+0000", 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.4429999999999999e-06 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181881+0000", 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.67285377899999999 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181891+0000", 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0479000000000001e-05 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.238 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.181961+0000", 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0370000000000006e-05 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:29.182566+0000", 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00060456400000000003 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "MOSDECSubOpRead(3.4s1 62/56 ECSubRead(tid=8, to_read={3:26eb5942:::smithi01639990-23:head=0,4096,0}, subchunks={3:26eb5942:::smithi01639990-23:head=[0,1]}, attrs_to_read=))", 2024-09-16T04:09:15.239 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 37.518716415, 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64312231099999995, 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.240 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647654+0000", 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.2500000000000004e-07 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:37.647678+0000", 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.3421000000000002e-05 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.241 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290437+0000", 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.64275884900000002 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290450+0000", 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.2858e-05 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290545+0000", 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.5463000000000004e-05 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:38.290776+0000", 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000230895 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:705 3.8s0 3:17b72cd6:::smithi01639990-47:head [setxattr _header (57) in=7b] snapc 18=[18,16,15,13] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.673629966, 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.037755205, 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.243 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 705 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.244 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492740+0000", 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492741+0000", 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.6199999999999997e-07 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492743+0000", 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7530000000000002e-06 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.245 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492750+0000", 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.7710000000000001e-06 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492756+0000", 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.524e-06 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494038+0000", 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0012823310000000001 2024-09-16T04:09:15.246 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494140+0000", 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010216400000000001 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_started", 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.494664+0000", 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00052402199999999996 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495429+0000", 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00076470399999999995 2024-09-16T04:09:15.247 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "commit_sent", 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530435+0000", 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035005738000000002 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530495+0000", 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0535999999999998e-05 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.248 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:706 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 26.673604242, 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.039505024, 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 706 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.249 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492766+0000", 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.0600000000000001e-07 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.250 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492769+0000", 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.7420000000000002e-06 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492776+0000", 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.0480000000000001e-06 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.492781+0000", 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1499999999999998e-06 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.251 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495503+0000", 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0027220199999999999 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.495628+0000", 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000125041 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530799+0000", 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.035171257999999997 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.252 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.530897+0000", 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.7608999999999993e-05 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:08:48.532271+0000", 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00137375 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:854 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~5554176,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:15.253 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.513773+0000", 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.652596607, 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025268701000000001, 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 854 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.254 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513773+0000", 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513773+0000", 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513775+0000", 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.04e-06 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.255 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513784+0000", 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.6459999999999999e-06 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513803+0000", 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.9177e-05 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513842+0000", 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 3.9273999999999997e-05 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.256 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513926+0000", 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.3793000000000002e-05 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "op must wait for map", 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513932+0000", 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.5589999999999996e-06 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.519949+0000", 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0060174740000000001 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.257 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520051+0000", 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.000102001 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.539042+0000", 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.018990737000000001 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:855 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~5554176,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.513882+0000", 2024-09-16T04:09:15.258 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.652487552, 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.032042586999999997, 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 855 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513882+0000", 2024-09-16T04:09:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513882+0000", 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513883+0000", 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0400000000000005e-07 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513891+0000", 2024-09-16T04:09:15.260 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.0269999999999995e-06 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513902+0000", 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 1.0572e-05 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513912+0000", 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.8579999999999995e-06 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.261 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514001+0000", 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.9176000000000001e-05 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514007+0000", 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.2230000000000004e-06 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520723+0000", 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0067159250000000002 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.262 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.520817+0000", 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.4265999999999995e-05 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.545925+0000", 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.025107635999999999 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4440.0:856 3.8s0 3:17b72cd6:::smithi01639990-47:head [read 0~5554176,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: "initiated_at": "2024-09-16T04:09:02.513928+0000", 2024-09-16T04:09:15.263 INFO:teuthology.orchestra.run.smithi194.stdout: "age": 12.652441958000001, 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.036141072000000003, 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "continuous": false, 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "started", 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4440", 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.16:0/3199945447", 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "tid": 856 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513928+0000", 2024-09-16T04:09:15.264 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513928+0000", 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "throttled", 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513929+0000", 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.7499999999999999e-07 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "all_read", 2024-09-16T04:09:15.265 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513933+0000", 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 4.5809999999999998e-06 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513941+0000", 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 7.8310000000000001e-06 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.513948+0000", 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.494e-06 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.266 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514041+0000", 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.3710999999999998e-05 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting_for_map not empty", 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.514047+0000", 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.0399999999999998e-06 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.521554+0000", 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0075068080000000002 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "started", 2024-09-16T04:09:15.267 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.521661+0000", 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.00010678799999999999 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: { 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2024-09-16T04:09:02.550069+0000", 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.028408044 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-16T04:09:15.268 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-16T04:09:15.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T04:09:15.389 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-16T04:09:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-16T04:09:15.389 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:15.398 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T04:09:15.506 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-16T04:09:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2024-09-16T04:09:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2024-09-16T04:09:15.506 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:15.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T04:09:15.623 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2024-09-16T04:09:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2024-09-16T04:09:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2024-09-16T04:09:15.623 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:15.756 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2024-09-16T04:09:15.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T04:09:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2024-09-16T04:09:17.456 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T04:09:17.456 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-16T04:09:17.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-09-16T04:09:17.771 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-09-16T04:09:17.770+0000 7f8c5c132640 -1 mon.a@0(leader).osd e98 definitely_dead 0 2024-09-16T04:09:18.772 INFO:teuthology.orchestra.run.smithi016.stderr:marked in osd.1. 2024-09-16T04:09:18.787 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-16T04:09:18.788 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T04:09:18.918 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a: {} 2024-09-16T04:09:18.918 INFO:teuthology.orchestra.run.smithi016.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:09:18.933 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b: {} 2024-09-16T04:09:18.934 INFO:teuthology.orchestra.run.smithi016.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:09:18.948 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c: {} 2024-09-16T04:09:18.948 INFO:teuthology.orchestra.run.smithi016.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T04:09:18.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T04:09:19.773 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T04:09:19.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-16T04:09:20.778 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T04:09:20.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T04:09:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T04:09:21.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T04:09:22.796 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T04:09:22.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T04:09:23.801 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T04:09:23.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-16T04:09:24.807 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T04:09:24.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T04:09:25.814 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T04:09:25.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-16T04:09:26.823 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T04:09:26.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T04:09:27.831 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T04:09:27.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-16T04:09:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T04:09:28.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T04:09:29.841 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T04:09:29.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-16T04:09:30.846 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T04:09:30.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T04:09:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T04:09:31.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-16T04:09:32.861 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T04:09:32.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T04:09:33.866 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T04:09:33.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-16T04:09:34.872 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-16T04:09:34.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T04:09:35.877 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T04:09:35.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-16T04:09:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-16T04:09:36.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T04:09:37.893 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T04:09:37.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-16T04:09:38.907 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-16T04:09:38.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T04:09:39.912 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T04:09:39.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-16T04:09:40.922 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-16T04:09:40.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T04:09:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T04:09:41.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T04:09:42.934 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T04:09:42.951 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T04:09:42.952 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T04:09:42.952 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T04:09:42.952 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T04:09:42.952 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T04:09:42.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:09:43.254 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:43.254 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":123,"fsid":"44381953-4509-4b15-853e-3887928dfa29","created":"2024-09-16T04:07:36.650572+0000","modified":"2024-09-16T04:09:42.931633+0000","last_up_change":"2024-09-16T04:08:51.432862+0000","last_in_change":"2024-09-16T04:09:18.766087+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:07:48.723300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:07:49.309634+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":0.75,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4506422f-7ccf-46d7-af66-62e8cace167a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6825","nonce":3110165165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6827","nonce":3110165165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6831","nonce":3110165165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6829","nonce":3110165165}]},"public_addr":"172.21.15.16:6825/3110165165","cluster_addr":"172.21.15.16:6827/3110165165","heartbeat_back_addr":"172.21.15.16:6831/3110165165","heartbeat_front_addr":"172.21.15.16:6829/3110165165","state":["exists","up"]},{"osd":1,"uuid":"8dc0488e-efb6-4664-a799-5e14c56769b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":54,"up_from":76,"up_thru":99,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6810","nonce":4190982331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6814","nonce":4190982331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6822","nonce":4190982331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6818","nonce":4190982331}]},"public_addr":"172.21.15.16:6810/4190982331","cluster_addr":"172.21.15.16:6814/4190982331","heartbeat_back_addr":"172.21.15.16:6822/4190982331","heartbeat_front_addr":"172.21.15.16:6818/4190982331","state":["exists","up"]},{"osd":2,"uuid":"1686e657-c1f3-499b-8b10-67f428c5c216","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6801","nonce":2266032822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6803","nonce":2266032822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6807","nonce":2266032822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6805","nonce":2266032822}]},"public_addr":"172.21.15.16:6801/2266032822","cluster_addr":"172.21.15.16:6803/2266032822","heartbeat_back_addr":"172.21.15.16:6807/2266032822","heartbeat_front_addr":"172.21.15.16:6805/2266032822","state":["exists","up"]},{"osd":3,"uuid":"730ad37c-1f5f-4d23-a1a7-c75320506665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6811","nonce":3389562884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6815","nonce":3389562884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6823","nonce":3389562884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6819","nonce":3389562884}]},"public_addr":"172.21.15.16:6811/3389562884","cluster_addr":"172.21.15.16:6815/3389562884","heartbeat_back_addr":"172.21.15.16:6823/3389562884","heartbeat_front_addr":"172.21.15.16:6819/3389562884","state":["exists","up"]},{"osd":4,"uuid":"d82c5c44-72fe-4680-9c04-ab4e0775b929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6811","nonce":1733996396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6813","nonce":1733996396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6817","nonce":1733996396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6815","nonce":1733996396}]},"public_addr":"172.21.15.174:6811/1733996396","cluster_addr":"172.21.15.174:6813/1733996396","heartbeat_back_addr":"172.21.15.174:6817/1733996396","heartbeat_front_addr":"172.21.15.174:6815/1733996396","state":["exists","up"]},{"osd":5,"uuid":"41502441-b0d9-460f-92c0-b8b3a915cac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6827","nonce":2956753716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6829","nonce":2956753716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6833","nonce":2956753716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6831","nonce":2956753716}]},"public_addr":"172.21.15.174:6827/2956753716","cluster_addr":"172.21.15.174:6829/2956753716","heartbeat_back_addr":"172.21.15.174:6833/2956753716","heartbeat_front_addr":"172.21.15.174:6831/2956753716","state":["exists","up"]},{"osd":6,"uuid":"45cc7521-11db-4496-8d0c-1abc42962810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6803","nonce":2727832023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6805","nonce":2727832023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6809","nonce":2727832023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6807","nonce":2727832023}]},"public_addr":"172.21.15.174:6803/2727832023","cluster_addr":"172.21.15.174:6805/2727832023","heartbeat_back_addr":"172.21.15.174:6809/2727832023","heartbeat_front_addr":"172.21.15.174:6807/2727832023","state":["exists","up"]},{"osd":7,"uuid":"1b5d6337-73c3-494b-91cd-fcfa7d307050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6819","nonce":2338392542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6821","nonce":2338392542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6825","nonce":2338392542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6823","nonce":2338392542}]},"public_addr":"172.21.15.174:6819/2338392542","cluster_addr":"172.21.15.174:6821/2338392542","heartbeat_back_addr":"172.21.15.174:6825/2338392542","heartbeat_front_addr":"172.21.15.174:6823/2338392542","state":["exists","up"]},{"osd":8,"uuid":"766efffa-3aff-4aa3-9cf8-88c5c7986d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621487148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621487148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621487148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621487148}]},"public_addr":"172.21.15.194:6801/2621487148","cluster_addr":"172.21.15.194:6803/2621487148","heartbeat_back_addr":"172.21.15.194:6807/2621487148","heartbeat_front_addr":"172.21.15.194:6805/2621487148","state":["exists","up"]},{"osd":9,"uuid":"755cc9aa-2f6f-4d7a-84cf-7feb74d65c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6817","nonce":367304498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6819","nonce":367304498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6823","nonce":367304498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6821","nonce":367304498}]},"public_addr":"172.21.15.194:6817/367304498","cluster_addr":"172.21.15.194:6819/367304498","heartbeat_back_addr":"172.21.15.194:6823/367304498","heartbeat_front_addr":"172.21.15.194:6821/367304498","state":["exists","up"]},{"osd":10,"uuid":"76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6825","nonce":3349503344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6827","nonce":3349503344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6831","nonce":3349503344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6829","nonce":3349503344}]},"public_addr":"172.21.15.194:6825/3349503344","cluster_addr":"172.21.15.194:6827/3349503344","heartbeat_back_addr":"172.21.15.194:6831/3349503344","heartbeat_front_addr":"172.21.15.194:6829/3349503344","state":["exists","up"]},{"osd":11,"uuid":"f71641eb-1331-4b39-9c0e-bf98e91ed295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6809","nonce":2531896090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6811","nonce":2531896090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6815","nonce":2531896090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6813","nonce":2531896090}]},"public_addr":"172.21.15.194:6809/2531896090","cluster_addr":"172.21.15.194:6811/2531896090","heartbeat_back_addr":"172.21.15.194:6815/2531896090","heartbeat_front_addr":"172.21.15.194:6813/2531896090","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:01.680673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:58.907614+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:47.992191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:48.214554+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:15.656396+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:59.631647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:58.489685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:08.502081+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:48.771857+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:06.156463+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:56.419990+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:00.745572+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":22,"length":1},{"begin":27,"length":1},{"begin":31,"length":1},{"begin":33,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:09:43.267 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T04:09:43.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:09:43.560 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:43.561 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":123,"fsid":"44381953-4509-4b15-853e-3887928dfa29","created":"2024-09-16T04:07:36.650572+0000","modified":"2024-09-16T04:09:42.931633+0000","last_up_change":"2024-09-16T04:08:51.432862+0000","last_in_change":"2024-09-16T04:09:18.766087+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:07:48.723300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:07:49.309634+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":0.75,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4506422f-7ccf-46d7-af66-62e8cace167a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6825","nonce":3110165165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6827","nonce":3110165165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6831","nonce":3110165165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6829","nonce":3110165165}]},"public_addr":"172.21.15.16:6825/3110165165","cluster_addr":"172.21.15.16:6827/3110165165","heartbeat_back_addr":"172.21.15.16:6831/3110165165","heartbeat_front_addr":"172.21.15.16:6829/3110165165","state":["exists","up"]},{"osd":1,"uuid":"8dc0488e-efb6-4664-a799-5e14c56769b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":54,"up_from":76,"up_thru":99,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6810","nonce":4190982331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6814","nonce":4190982331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6822","nonce":4190982331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6818","nonce":4190982331}]},"public_addr":"172.21.15.16:6810/4190982331","cluster_addr":"172.21.15.16:6814/4190982331","heartbeat_back_addr":"172.21.15.16:6822/4190982331","heartbeat_front_addr":"172.21.15.16:6818/4190982331","state":["exists","up"]},{"osd":2,"uuid":"1686e657-c1f3-499b-8b10-67f428c5c216","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6801","nonce":2266032822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6803","nonce":2266032822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6807","nonce":2266032822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6805","nonce":2266032822}]},"public_addr":"172.21.15.16:6801/2266032822","cluster_addr":"172.21.15.16:6803/2266032822","heartbeat_back_addr":"172.21.15.16:6807/2266032822","heartbeat_front_addr":"172.21.15.16:6805/2266032822","state":["exists","up"]},{"osd":3,"uuid":"730ad37c-1f5f-4d23-a1a7-c75320506665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6811","nonce":3389562884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6815","nonce":3389562884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6823","nonce":3389562884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6819","nonce":3389562884}]},"public_addr":"172.21.15.16:6811/3389562884","cluster_addr":"172.21.15.16:6815/3389562884","heartbeat_back_addr":"172.21.15.16:6823/3389562884","heartbeat_front_addr":"172.21.15.16:6819/3389562884","state":["exists","up"]},{"osd":4,"uuid":"d82c5c44-72fe-4680-9c04-ab4e0775b929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6811","nonce":1733996396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6813","nonce":1733996396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6817","nonce":1733996396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6815","nonce":1733996396}]},"public_addr":"172.21.15.174:6811/1733996396","cluster_addr":"172.21.15.174:6813/1733996396","heartbeat_back_addr":"172.21.15.174:6817/1733996396","heartbeat_front_addr":"172.21.15.174:6815/1733996396","state":["exists","up"]},{"osd":5,"uuid":"41502441-b0d9-460f-92c0-b8b3a915cac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6827","nonce":2956753716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6829","nonce":2956753716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6833","nonce":2956753716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6831","nonce":2956753716}]},"public_addr":"172.21.15.174:6827/2956753716","cluster_addr":"172.21.15.174:6829/2956753716","heartbeat_back_addr":"172.21.15.174:6833/2956753716","heartbeat_front_addr":"172.21.15.174:6831/2956753716","state":["exists","up"]},{"osd":6,"uuid":"45cc7521-11db-4496-8d0c-1abc42962810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6803","nonce":2727832023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6805","nonce":2727832023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6809","nonce":2727832023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6807","nonce":2727832023}]},"public_addr":"172.21.15.174:6803/2727832023","cluster_addr":"172.21.15.174:6805/2727832023","heartbeat_back_addr":"172.21.15.174:6809/2727832023","heartbeat_front_addr":"172.21.15.174:6807/2727832023","state":["exists","up"]},{"osd":7,"uuid":"1b5d6337-73c3-494b-91cd-fcfa7d307050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6819","nonce":2338392542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6821","nonce":2338392542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6825","nonce":2338392542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6823","nonce":2338392542}]},"public_addr":"172.21.15.174:6819/2338392542","cluster_addr":"172.21.15.174:6821/2338392542","heartbeat_back_addr":"172.21.15.174:6825/2338392542","heartbeat_front_addr":"172.21.15.174:6823/2338392542","state":["exists","up"]},{"osd":8,"uuid":"766efffa-3aff-4aa3-9cf8-88c5c7986d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621487148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621487148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621487148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621487148}]},"public_addr":"172.21.15.194:6801/2621487148","cluster_addr":"172.21.15.194:6803/2621487148","heartbeat_back_addr":"172.21.15.194:6807/2621487148","heartbeat_front_addr":"172.21.15.194:6805/2621487148","state":["exists","up"]},{"osd":9,"uuid":"755cc9aa-2f6f-4d7a-84cf-7feb74d65c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6817","nonce":367304498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6819","nonce":367304498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6823","nonce":367304498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6821","nonce":367304498}]},"public_addr":"172.21.15.194:6817/367304498","cluster_addr":"172.21.15.194:6819/367304498","heartbeat_back_addr":"172.21.15.194:6823/367304498","heartbeat_front_addr":"172.21.15.194:6821/367304498","state":["exists","up"]},{"osd":10,"uuid":"76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6825","nonce":3349503344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6827","nonce":3349503344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6831","nonce":3349503344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6829","nonce":3349503344}]},"public_addr":"172.21.15.194:6825/3349503344","cluster_addr":"172.21.15.194:6827/3349503344","heartbeat_back_addr":"172.21.15.194:6831/3349503344","heartbeat_front_addr":"172.21.15.194:6829/3349503344","state":["exists","up"]},{"osd":11,"uuid":"f71641eb-1331-4b39-9c0e-bf98e91ed295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6809","nonce":2531896090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6811","nonce":2531896090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6815","nonce":2531896090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6813","nonce":2531896090}]},"public_addr":"172.21.15.194:6809/2531896090","cluster_addr":"172.21.15.194:6811/2531896090","heartbeat_back_addr":"172.21.15.194:6815/2531896090","heartbeat_front_addr":"172.21.15.194:6813/2531896090","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:01.680673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:58.907614+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:47.992191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:48.214554+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:15.656396+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:59.631647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:58.489685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:08.502081+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:48.771857+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:06.156463+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:56.419990+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:00.745572+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":22,"length":1},{"begin":27,"length":1},{"begin":31,"length":1},{"begin":33,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:09:43.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:09:43.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:09:43.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:09:43.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:09:43.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T04:09:43.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T04:09:43.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T04:09:43.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T04:09:43.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T04:09:43.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T04:09:43.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T04:09:43.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T04:09:43.914 INFO:teuthology.orchestra.run.smithi016.stdout:81604378652 2024-09-16T04:09:43.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:09:43.926 INFO:teuthology.orchestra.run.smithi016.stdout:326417514508 2024-09-16T04:09:43.926 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:09:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:73014444059 2024-09-16T04:09:43.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:09:43.936 INFO:teuthology.orchestra.run.smithi016.stdout:81604378649 2024-09-16T04:09:43.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:09:43.943 INFO:teuthology.orchestra.run.smithi016.stdout:77309411353 2024-09-16T04:09:43.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:09:43.949 INFO:teuthology.orchestra.run.smithi016.stdout:77309411355 2024-09-16T04:09:43.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:09:43.959 INFO:teuthology.orchestra.run.smithi016.stdout:77309411354 2024-09-16T04:09:43.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:09:43.962 INFO:teuthology.orchestra.run.smithi016.stdout:81604378650 2024-09-16T04:09:43.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:09:43.966 INFO:teuthology.orchestra.run.smithi016.stdout:81604378649 2024-09-16T04:09:43.967 INFO:teuthology.orchestra.run.smithi016.stdout:81604378650 2024-09-16T04:09:43.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:09:43.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:09:43.974 INFO:teuthology.orchestra.run.smithi016.stdout:77309411356 2024-09-16T04:09:43.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:09:43.977 INFO:teuthology.orchestra.run.smithi016.stdout:77309411354 2024-09-16T04:09:43.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:09:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:77309411354 2024-09-16T04:09:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:73014444058 2024-09-16T04:09:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:81604378651 2024-09-16T04:09:44.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411355 got 77309411354 for osd.7 2024-09-16T04:09:44.746 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378652 got 81604378651 for osd.3 2024-09-16T04:09:44.757 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444059 got 73014444058 for osd.0 2024-09-16T04:09:44.782 INFO:teuthology.orchestra.run.smithi016.stdout:81604378648 2024-09-16T04:09:44.795 INFO:teuthology.orchestra.run.smithi016.stdout:326417514507 2024-09-16T04:09:44.819 INFO:teuthology.orchestra.run.smithi016.stdout:77309411352 2024-09-16T04:09:44.821 INFO:tasks.ceph.ceph_manager.ceph:need seq 326417514508 got 326417514507 for osd.1 2024-09-16T04:09:44.827 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378649 got 81604378648 for osd.5 2024-09-16T04:09:44.832 INFO:teuthology.orchestra.run.smithi016.stdout:77309411352 2024-09-16T04:09:44.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411353 got 77309411352 for osd.8 2024-09-16T04:09:44.847 INFO:teuthology.orchestra.run.smithi016.stdout:81604378649 2024-09-16T04:09:44.854 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411354 got 77309411352 for osd.11 2024-09-16T04:09:44.859 INFO:teuthology.orchestra.run.smithi016.stdout:77309411354 2024-09-16T04:09:44.862 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378650 got 81604378649 for osd.2 2024-09-16T04:09:44.873 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411356 got 77309411354 for osd.10 2024-09-16T04:09:44.877 INFO:teuthology.orchestra.run.smithi016.stdout:81604378649 2024-09-16T04:09:44.886 INFO:teuthology.orchestra.run.smithi016.stdout:81604378648 2024-09-16T04:09:44.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378650 got 81604378649 for osd.6 2024-09-16T04:09:44.899 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378649 got 81604378648 for osd.4 2024-09-16T04:09:44.921 INFO:teuthology.orchestra.run.smithi016.stdout:77309411352 2024-09-16T04:09:44.933 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411354 got 77309411352 for osd.9 2024-09-16T04:09:45.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T04:09:45.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:09:45.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:09:45.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:09:45.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:09:45.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T04:09:45.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T04:09:45.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:09:45.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T04:09:45.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T04:09:45.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:09:45.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T04:09:46.318 INFO:teuthology.orchestra.run.smithi016.stdout:77309411356 2024-09-16T04:09:46.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411355 got 77309411356 for osd.7 2024-09-16T04:09:46.350 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.380 INFO:teuthology.orchestra.run.smithi016.stdout:81604378652 2024-09-16T04:09:46.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378652 got 81604378652 for osd.3 2024-09-16T04:09:46.410 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.444 INFO:teuthology.orchestra.run.smithi016.stdout:73014444059 2024-09-16T04:09:46.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444059 got 73014444059 for osd.0 2024-09-16T04:09:46.480 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.573 INFO:teuthology.orchestra.run.smithi016.stdout:326417514508 2024-09-16T04:09:46.597 INFO:teuthology.orchestra.run.smithi016.stdout:77309411354 2024-09-16T04:09:46.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 326417514508 got 326417514508 for osd.1 2024-09-16T04:09:46.598 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411354 got 77309411354 for osd.11 2024-09-16T04:09:46.615 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.622 INFO:teuthology.orchestra.run.smithi016.stdout:81604378650 2024-09-16T04:09:46.635 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378650 got 81604378650 for osd.2 2024-09-16T04:09:46.636 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.652 INFO:teuthology.orchestra.run.smithi016.stdout:81604378649 2024-09-16T04:09:46.666 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378649 got 81604378649 for osd.5 2024-09-16T04:09:46.666 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.682 INFO:teuthology.orchestra.run.smithi016.stdout:77309411354 2024-09-16T04:09:46.683 INFO:teuthology.orchestra.run.smithi016.stdout:77309411354 2024-09-16T04:09:46.701 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411354 got 77309411354 for osd.9 2024-09-16T04:09:46.702 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.702 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411353 got 77309411354 for osd.8 2024-09-16T04:09:46.702 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.725 INFO:teuthology.orchestra.run.smithi016.stdout:81604378650 2024-09-16T04:09:46.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378650 got 81604378650 for osd.6 2024-09-16T04:09:46.739 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.745 INFO:teuthology.orchestra.run.smithi016.stdout:77309411356 2024-09-16T04:09:46.763 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411356 got 77309411356 for osd.10 2024-09-16T04:09:46.763 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.790 INFO:teuthology.orchestra.run.smithi016.stdout:81604378649 2024-09-16T04:09:46.803 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378649 got 81604378649 for osd.4 2024-09-16T04:09:46.803 DEBUG:teuthology.parallel:result is None 2024-09-16T04:09:46.803 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T04:09:46.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:09:47.138 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:47.139 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:09:47.150 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-16T04:09:46.692761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":540096,"kb_used_data":18028,"kb_used_omap":19,"kb_used_meta":521772,"kb_avail":1131921984,"statfs":{"total":1159641169920,"available":1159088111616,"internally_reserved":0,"allocated":18460672,"data_stored":11956222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20453,"internal_metadata":534294555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001345"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":225,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.936638+0000","last_change":"2024-09-16T04:09:08.905295+0000","last_active":"2024-09-16T04:09:42.936638+0000","last_peered":"2024-09-16T04:09:42.936638+0000","last_clean":"2024-09-16T04:09:42.936638+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:09:42.936638+0000","last_undegraded":"2024-09-16T04:09:42.936638+0000","last_fullsized":"2024-09-16T04:09:42.936638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:08.905173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:08.905173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:32.060373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691311+0000","last_change":"2024-09-16T04:09:19.842195+0000","last_active":"2024-09-16T04:09:43.691311+0000","last_peered":"2024-09-16T04:09:43.691311+0000","last_clean":"2024-09-16T04:09:43.691311+0000","last_became_active":"2024-09-16T04:09:19.841940+0000","last_became_peered":"2024-09-16T04:09:19.841940+0000","last_unstale":"2024-09-16T04:09:43.691311+0000","last_undegraded":"2024-09-16T04:09:43.691311+0000","last_fullsized":"2024-09-16T04:09:43.691311+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:09.860925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:09.860925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:22.770968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.593889+0000","last_change":"2024-09-16T04:09:16.871123+0000","last_active":"2024-09-16T04:09:43.593889+0000","last_peered":"2024-09-16T04:09:43.593889+0000","last_clean":"2024-09-16T04:09:43.593889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:09:43.593889+0000","last_undegraded":"2024-09-16T04:09:43.593889+0000","last_fullsized":"2024-09-16T04:09:43.593889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:16.871004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:16.871004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.338451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691194+0000","last_change":"2024-09-16T04:09:19.780917+0000","last_active":"2024-09-16T04:09:43.691194+0000","last_peered":"2024-09-16T04:09:43.691194+0000","last_clean":"2024-09-16T04:09:43.691194+0000","last_became_active":"2024-09-16T04:09:19.780560+0000","last_became_peered":"2024-09-16T04:09:19.780560+0000","last_unstale":"2024-09-16T04:09:43.691194+0000","last_undegraded":"2024-09-16T04:09:43.691194+0000","last_fullsized":"2024-09-16T04:09:43.691194+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:54.852438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:54.852438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:59.574073+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"84'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937555+0000","last_change":"2024-09-16T04:09:19.779732+0000","last_active":"2024-09-16T04:09:42.937555+0000","last_peered":"2024-09-16T04:09:42.937555+0000","last_clean":"2024-09-16T04:09:42.937555+0000","last_became_active":"2024-09-16T04:09:19.776967+0000","last_became_peered":"2024-09-16T04:09:19.776967+0000","last_unstale":"2024-09-16T04:09:42.937555+0000","last_undegraded":"2024-09-16T04:09:42.937555+0000","last_fullsized":"2024-09-16T04:09:42.937555+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3","last_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_deep_scrub":"84'3","last_deep_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_clean_scrub_stamp":"2024-09-16T04:09:00.126257+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:07.593035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":224,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.358406+0000","last_change":"2024-09-16T04:09:22.362739+0000","last_active":"2024-09-16T04:09:43.358406+0000","last_peered":"2024-09-16T04:09:43.358406+0000","last_clean":"2024-09-16T04:09:43.358406+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:09:43.358406+0000","last_undegraded":"2024-09-16T04:09:43.358406+0000","last_fullsized":"2024-09-16T04:09:43.358406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:22.362652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:22.362652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.113016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691276+0000","last_change":"2024-09-16T04:08:58.876711+0000","last_active":"2024-09-16T04:09:43.691276+0000","last_peered":"2024-09-16T04:09:43.691276+0000","last_clean":"2024-09-16T04:09:43.691276+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:09:43.691276+0000","last_undegraded":"2024-09-16T04:09:43.691276+0000","last_fullsized":"2024-09-16T04:09:43.691276+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_clean_scrub_stamp":"2024-09-16T04:08:58.876583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:28.287717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":227,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937570+0000","last_change":"2024-09-16T04:08:59.147922+0000","last_active":"2024-09-16T04:09:42.937570+0000","last_peered":"2024-09-16T04:09:42.937570+0000","last_clean":"2024-09-16T04:09:42.937570+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:09:42.937570+0000","last_undegraded":"2024-09-16T04:09:42.937570+0000","last_fullsized":"2024-09-16T04:09:42.937570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-09-16T04:08:59.147797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:59.147797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:25.871032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":392,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691286+0000","last_change":"2024-09-16T04:08:55.861559+0000","last_active":"2024-09-16T04:09:43.691286+0000","last_peered":"2024-09-16T04:09:43.691286+0000","last_clean":"2024-09-16T04:09:43.691286+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:09:43.691286+0000","last_undegraded":"2024-09-16T04:09:43.691286+0000","last_fullsized":"2024-09-16T04:09:43.691286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'137","last_scrub_stamp":"2024-09-16T04:08:55.861468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:08:55.861468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:57.766582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378651,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50660,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321180,"statfs":{"total":96636764160,"available":96584888320,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54820,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317020,"statfs":{"total":96636764160,"available":96580628480,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378650,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50656,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378653,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58976,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312864,"statfs":{"total":96636764160,"available":96576372736,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411354,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55324,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316516,"statfs":{"total":96636764160,"available":96580112384,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50652,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321188,"statfs":{"total":96636764160,"available":96584896512,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51164,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320676,"statfs":{"total":96636764160,"available":96584372224,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":76,"seq":326417514509,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":1424,"kb_used_omap":2,"kb_used_meta":27645,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2908,"internal_metadata":28308644},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444059,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:09:47.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:09:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:47.445 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:09:47.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-16T04:09:46.692761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":540096,"kb_used_data":18028,"kb_used_omap":19,"kb_used_meta":521772,"kb_avail":1131921984,"statfs":{"total":1159641169920,"available":1159088111616,"internally_reserved":0,"allocated":18460672,"data_stored":11956222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20453,"internal_metadata":534294555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001345"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":225,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.936638+0000","last_change":"2024-09-16T04:09:08.905295+0000","last_active":"2024-09-16T04:09:42.936638+0000","last_peered":"2024-09-16T04:09:42.936638+0000","last_clean":"2024-09-16T04:09:42.936638+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:09:42.936638+0000","last_undegraded":"2024-09-16T04:09:42.936638+0000","last_fullsized":"2024-09-16T04:09:42.936638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:08.905173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:08.905173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:32.060373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691311+0000","last_change":"2024-09-16T04:09:19.842195+0000","last_active":"2024-09-16T04:09:43.691311+0000","last_peered":"2024-09-16T04:09:43.691311+0000","last_clean":"2024-09-16T04:09:43.691311+0000","last_became_active":"2024-09-16T04:09:19.841940+0000","last_became_peered":"2024-09-16T04:09:19.841940+0000","last_unstale":"2024-09-16T04:09:43.691311+0000","last_undegraded":"2024-09-16T04:09:43.691311+0000","last_fullsized":"2024-09-16T04:09:43.691311+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:09.860925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:09.860925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:22.770968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.593889+0000","last_change":"2024-09-16T04:09:16.871123+0000","last_active":"2024-09-16T04:09:43.593889+0000","last_peered":"2024-09-16T04:09:43.593889+0000","last_clean":"2024-09-16T04:09:43.593889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:09:43.593889+0000","last_undegraded":"2024-09-16T04:09:43.593889+0000","last_fullsized":"2024-09-16T04:09:43.593889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:16.871004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:16.871004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.338451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691194+0000","last_change":"2024-09-16T04:09:19.780917+0000","last_active":"2024-09-16T04:09:43.691194+0000","last_peered":"2024-09-16T04:09:43.691194+0000","last_clean":"2024-09-16T04:09:43.691194+0000","last_became_active":"2024-09-16T04:09:19.780560+0000","last_became_peered":"2024-09-16T04:09:19.780560+0000","last_unstale":"2024-09-16T04:09:43.691194+0000","last_undegraded":"2024-09-16T04:09:43.691194+0000","last_fullsized":"2024-09-16T04:09:43.691194+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:54.852438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:54.852438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:59.574073+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"84'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937555+0000","last_change":"2024-09-16T04:09:19.779732+0000","last_active":"2024-09-16T04:09:42.937555+0000","last_peered":"2024-09-16T04:09:42.937555+0000","last_clean":"2024-09-16T04:09:42.937555+0000","last_became_active":"2024-09-16T04:09:19.776967+0000","last_became_peered":"2024-09-16T04:09:19.776967+0000","last_unstale":"2024-09-16T04:09:42.937555+0000","last_undegraded":"2024-09-16T04:09:42.937555+0000","last_fullsized":"2024-09-16T04:09:42.937555+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3","last_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_deep_scrub":"84'3","last_deep_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_clean_scrub_stamp":"2024-09-16T04:09:00.126257+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:07.593035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":224,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.358406+0000","last_change":"2024-09-16T04:09:22.362739+0000","last_active":"2024-09-16T04:09:43.358406+0000","last_peered":"2024-09-16T04:09:43.358406+0000","last_clean":"2024-09-16T04:09:43.358406+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:09:43.358406+0000","last_undegraded":"2024-09-16T04:09:43.358406+0000","last_fullsized":"2024-09-16T04:09:43.358406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:22.362652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:22.362652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.113016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691276+0000","last_change":"2024-09-16T04:08:58.876711+0000","last_active":"2024-09-16T04:09:43.691276+0000","last_peered":"2024-09-16T04:09:43.691276+0000","last_clean":"2024-09-16T04:09:43.691276+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:09:43.691276+0000","last_undegraded":"2024-09-16T04:09:43.691276+0000","last_fullsized":"2024-09-16T04:09:43.691276+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_clean_scrub_stamp":"2024-09-16T04:08:58.876583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:28.287717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":227,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937570+0000","last_change":"2024-09-16T04:08:59.147922+0000","last_active":"2024-09-16T04:09:42.937570+0000","last_peered":"2024-09-16T04:09:42.937570+0000","last_clean":"2024-09-16T04:09:42.937570+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:09:42.937570+0000","last_undegraded":"2024-09-16T04:09:42.937570+0000","last_fullsized":"2024-09-16T04:09:42.937570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-09-16T04:08:59.147797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:59.147797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:25.871032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":392,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691286+0000","last_change":"2024-09-16T04:08:55.861559+0000","last_active":"2024-09-16T04:09:43.691286+0000","last_peered":"2024-09-16T04:09:43.691286+0000","last_clean":"2024-09-16T04:09:43.691286+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:09:43.691286+0000","last_undegraded":"2024-09-16T04:09:43.691286+0000","last_fullsized":"2024-09-16T04:09:43.691286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'137","last_scrub_stamp":"2024-09-16T04:08:55.861468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:08:55.861468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:57.766582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378651,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50660,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321180,"statfs":{"total":96636764160,"available":96584888320,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54820,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317020,"statfs":{"total":96636764160,"available":96580628480,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378650,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50656,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378653,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58976,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312864,"statfs":{"total":96636764160,"available":96576372736,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411354,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55324,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316516,"statfs":{"total":96636764160,"available":96580112384,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50652,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321188,"statfs":{"total":96636764160,"available":96584896512,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51164,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320676,"statfs":{"total":96636764160,"available":96584372224,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":76,"seq":326417514509,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":1424,"kb_used_omap":2,"kb_used_meta":27645,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2908,"internal_metadata":28308644},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444059,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:09:47.458 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T04:09:47.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-16T04:09:47.769 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T04:09:47.778 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:09:47.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:09:48.068 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:48.068 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:09:48.080 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-16T04:09:46.692761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":540096,"kb_used_data":18028,"kb_used_omap":19,"kb_used_meta":521772,"kb_avail":1131921984,"statfs":{"total":1159641169920,"available":1159088111616,"internally_reserved":0,"allocated":18460672,"data_stored":11956222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20453,"internal_metadata":534294555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001345"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":225,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.936638+0000","last_change":"2024-09-16T04:09:08.905295+0000","last_active":"2024-09-16T04:09:42.936638+0000","last_peered":"2024-09-16T04:09:42.936638+0000","last_clean":"2024-09-16T04:09:42.936638+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:09:42.936638+0000","last_undegraded":"2024-09-16T04:09:42.936638+0000","last_fullsized":"2024-09-16T04:09:42.936638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:08.905173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:08.905173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:32.060373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691311+0000","last_change":"2024-09-16T04:09:19.842195+0000","last_active":"2024-09-16T04:09:43.691311+0000","last_peered":"2024-09-16T04:09:43.691311+0000","last_clean":"2024-09-16T04:09:43.691311+0000","last_became_active":"2024-09-16T04:09:19.841940+0000","last_became_peered":"2024-09-16T04:09:19.841940+0000","last_unstale":"2024-09-16T04:09:43.691311+0000","last_undegraded":"2024-09-16T04:09:43.691311+0000","last_fullsized":"2024-09-16T04:09:43.691311+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:09.860925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:09.860925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:22.770968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.593889+0000","last_change":"2024-09-16T04:09:16.871123+0000","last_active":"2024-09-16T04:09:43.593889+0000","last_peered":"2024-09-16T04:09:43.593889+0000","last_clean":"2024-09-16T04:09:43.593889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:09:43.593889+0000","last_undegraded":"2024-09-16T04:09:43.593889+0000","last_fullsized":"2024-09-16T04:09:43.593889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:16.871004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:16.871004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.338451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691194+0000","last_change":"2024-09-16T04:09:19.780917+0000","last_active":"2024-09-16T04:09:43.691194+0000","last_peered":"2024-09-16T04:09:43.691194+0000","last_clean":"2024-09-16T04:09:43.691194+0000","last_became_active":"2024-09-16T04:09:19.780560+0000","last_became_peered":"2024-09-16T04:09:19.780560+0000","last_unstale":"2024-09-16T04:09:43.691194+0000","last_undegraded":"2024-09-16T04:09:43.691194+0000","last_fullsized":"2024-09-16T04:09:43.691194+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:54.852438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:54.852438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:59.574073+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"84'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937555+0000","last_change":"2024-09-16T04:09:19.779732+0000","last_active":"2024-09-16T04:09:42.937555+0000","last_peered":"2024-09-16T04:09:42.937555+0000","last_clean":"2024-09-16T04:09:42.937555+0000","last_became_active":"2024-09-16T04:09:19.776967+0000","last_became_peered":"2024-09-16T04:09:19.776967+0000","last_unstale":"2024-09-16T04:09:42.937555+0000","last_undegraded":"2024-09-16T04:09:42.937555+0000","last_fullsized":"2024-09-16T04:09:42.937555+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3","last_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_deep_scrub":"84'3","last_deep_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_clean_scrub_stamp":"2024-09-16T04:09:00.126257+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:07.593035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":224,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.358406+0000","last_change":"2024-09-16T04:09:22.362739+0000","last_active":"2024-09-16T04:09:43.358406+0000","last_peered":"2024-09-16T04:09:43.358406+0000","last_clean":"2024-09-16T04:09:43.358406+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:09:43.358406+0000","last_undegraded":"2024-09-16T04:09:43.358406+0000","last_fullsized":"2024-09-16T04:09:43.358406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:22.362652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:22.362652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.113016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691276+0000","last_change":"2024-09-16T04:08:58.876711+0000","last_active":"2024-09-16T04:09:43.691276+0000","last_peered":"2024-09-16T04:09:43.691276+0000","last_clean":"2024-09-16T04:09:43.691276+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:09:43.691276+0000","last_undegraded":"2024-09-16T04:09:43.691276+0000","last_fullsized":"2024-09-16T04:09:43.691276+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_clean_scrub_stamp":"2024-09-16T04:08:58.876583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:28.287717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":227,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937570+0000","last_change":"2024-09-16T04:08:59.147922+0000","last_active":"2024-09-16T04:09:42.937570+0000","last_peered":"2024-09-16T04:09:42.937570+0000","last_clean":"2024-09-16T04:09:42.937570+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:09:42.937570+0000","last_undegraded":"2024-09-16T04:09:42.937570+0000","last_fullsized":"2024-09-16T04:09:42.937570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-09-16T04:08:59.147797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:59.147797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:25.871032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":392,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691286+0000","last_change":"2024-09-16T04:08:55.861559+0000","last_active":"2024-09-16T04:09:43.691286+0000","last_peered":"2024-09-16T04:09:43.691286+0000","last_clean":"2024-09-16T04:09:43.691286+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:09:43.691286+0000","last_undegraded":"2024-09-16T04:09:43.691286+0000","last_fullsized":"2024-09-16T04:09:43.691286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'137","last_scrub_stamp":"2024-09-16T04:08:55.861468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:08:55.861468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:57.766582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378651,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50660,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321180,"statfs":{"total":96636764160,"available":96584888320,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54820,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317020,"statfs":{"total":96636764160,"available":96580628480,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378650,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50656,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378653,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58976,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312864,"statfs":{"total":96636764160,"available":96576372736,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411354,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55324,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316516,"statfs":{"total":96636764160,"available":96580112384,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50652,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321188,"statfs":{"total":96636764160,"available":96584896512,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51164,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320676,"statfs":{"total":96636764160,"available":96584372224,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":76,"seq":326417514509,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":1424,"kb_used_omap":2,"kb_used_meta":27645,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2908,"internal_metadata":28308644},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444059,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:09:48.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:09:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:48.375 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:09:48.388 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-16T04:09:46.692761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":540096,"kb_used_data":18028,"kb_used_omap":19,"kb_used_meta":521772,"kb_avail":1131921984,"statfs":{"total":1159641169920,"available":1159088111616,"internally_reserved":0,"allocated":18460672,"data_stored":11956222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20453,"internal_metadata":534294555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001345"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":225,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.936638+0000","last_change":"2024-09-16T04:09:08.905295+0000","last_active":"2024-09-16T04:09:42.936638+0000","last_peered":"2024-09-16T04:09:42.936638+0000","last_clean":"2024-09-16T04:09:42.936638+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:09:42.936638+0000","last_undegraded":"2024-09-16T04:09:42.936638+0000","last_fullsized":"2024-09-16T04:09:42.936638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:08.905173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:08.905173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:32.060373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691311+0000","last_change":"2024-09-16T04:09:19.842195+0000","last_active":"2024-09-16T04:09:43.691311+0000","last_peered":"2024-09-16T04:09:43.691311+0000","last_clean":"2024-09-16T04:09:43.691311+0000","last_became_active":"2024-09-16T04:09:19.841940+0000","last_became_peered":"2024-09-16T04:09:19.841940+0000","last_unstale":"2024-09-16T04:09:43.691311+0000","last_undegraded":"2024-09-16T04:09:43.691311+0000","last_fullsized":"2024-09-16T04:09:43.691311+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:09.860925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:09.860925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:22.770968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.593889+0000","last_change":"2024-09-16T04:09:16.871123+0000","last_active":"2024-09-16T04:09:43.593889+0000","last_peered":"2024-09-16T04:09:43.593889+0000","last_clean":"2024-09-16T04:09:43.593889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:09:43.593889+0000","last_undegraded":"2024-09-16T04:09:43.593889+0000","last_fullsized":"2024-09-16T04:09:43.593889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:16.871004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:16.871004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.338451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691194+0000","last_change":"2024-09-16T04:09:19.780917+0000","last_active":"2024-09-16T04:09:43.691194+0000","last_peered":"2024-09-16T04:09:43.691194+0000","last_clean":"2024-09-16T04:09:43.691194+0000","last_became_active":"2024-09-16T04:09:19.780560+0000","last_became_peered":"2024-09-16T04:09:19.780560+0000","last_unstale":"2024-09-16T04:09:43.691194+0000","last_undegraded":"2024-09-16T04:09:43.691194+0000","last_fullsized":"2024-09-16T04:09:43.691194+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:54.852438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:54.852438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:59.574073+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"84'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937555+0000","last_change":"2024-09-16T04:09:19.779732+0000","last_active":"2024-09-16T04:09:42.937555+0000","last_peered":"2024-09-16T04:09:42.937555+0000","last_clean":"2024-09-16T04:09:42.937555+0000","last_became_active":"2024-09-16T04:09:19.776967+0000","last_became_peered":"2024-09-16T04:09:19.776967+0000","last_unstale":"2024-09-16T04:09:42.937555+0000","last_undegraded":"2024-09-16T04:09:42.937555+0000","last_fullsized":"2024-09-16T04:09:42.937555+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3","last_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_deep_scrub":"84'3","last_deep_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_clean_scrub_stamp":"2024-09-16T04:09:00.126257+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:07.593035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":224,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.358406+0000","last_change":"2024-09-16T04:09:22.362739+0000","last_active":"2024-09-16T04:09:43.358406+0000","last_peered":"2024-09-16T04:09:43.358406+0000","last_clean":"2024-09-16T04:09:43.358406+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:09:43.358406+0000","last_undegraded":"2024-09-16T04:09:43.358406+0000","last_fullsized":"2024-09-16T04:09:43.358406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:22.362652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:22.362652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.113016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691276+0000","last_change":"2024-09-16T04:08:58.876711+0000","last_active":"2024-09-16T04:09:43.691276+0000","last_peered":"2024-09-16T04:09:43.691276+0000","last_clean":"2024-09-16T04:09:43.691276+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:09:43.691276+0000","last_undegraded":"2024-09-16T04:09:43.691276+0000","last_fullsized":"2024-09-16T04:09:43.691276+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_clean_scrub_stamp":"2024-09-16T04:08:58.876583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:28.287717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":227,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937570+0000","last_change":"2024-09-16T04:08:59.147922+0000","last_active":"2024-09-16T04:09:42.937570+0000","last_peered":"2024-09-16T04:09:42.937570+0000","last_clean":"2024-09-16T04:09:42.937570+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:09:42.937570+0000","last_undegraded":"2024-09-16T04:09:42.937570+0000","last_fullsized":"2024-09-16T04:09:42.937570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-09-16T04:08:59.147797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:59.147797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:25.871032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":392,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691286+0000","last_change":"2024-09-16T04:08:55.861559+0000","last_active":"2024-09-16T04:09:43.691286+0000","last_peered":"2024-09-16T04:09:43.691286+0000","last_clean":"2024-09-16T04:09:43.691286+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:09:43.691286+0000","last_undegraded":"2024-09-16T04:09:43.691286+0000","last_fullsized":"2024-09-16T04:09:43.691286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'137","last_scrub_stamp":"2024-09-16T04:08:55.861468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:08:55.861468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:57.766582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378651,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50660,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321180,"statfs":{"total":96636764160,"available":96584888320,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54820,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317020,"statfs":{"total":96636764160,"available":96580628480,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378650,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50656,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378653,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58976,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312864,"statfs":{"total":96636764160,"available":96576372736,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411354,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55324,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316516,"statfs":{"total":96636764160,"available":96580112384,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50652,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321188,"statfs":{"total":96636764160,"available":96584896512,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51164,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320676,"statfs":{"total":96636764160,"available":96584372224,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":76,"seq":326417514509,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":1424,"kb_used_omap":2,"kb_used_meta":27645,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2908,"internal_metadata":28308644},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444059,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:09:48.389 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:09:48.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:09:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:48.679 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:09:48.691 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-16T04:09:46.692761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":540096,"kb_used_data":18028,"kb_used_omap":19,"kb_used_meta":521772,"kb_avail":1131921984,"statfs":{"total":1159641169920,"available":1159088111616,"internally_reserved":0,"allocated":18460672,"data_stored":11956222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20453,"internal_metadata":534294555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001345"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":225,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.936638+0000","last_change":"2024-09-16T04:09:08.905295+0000","last_active":"2024-09-16T04:09:42.936638+0000","last_peered":"2024-09-16T04:09:42.936638+0000","last_clean":"2024-09-16T04:09:42.936638+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:09:42.936638+0000","last_undegraded":"2024-09-16T04:09:42.936638+0000","last_fullsized":"2024-09-16T04:09:42.936638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:08.905173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:08.905173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:32.060373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691311+0000","last_change":"2024-09-16T04:09:19.842195+0000","last_active":"2024-09-16T04:09:43.691311+0000","last_peered":"2024-09-16T04:09:43.691311+0000","last_clean":"2024-09-16T04:09:43.691311+0000","last_became_active":"2024-09-16T04:09:19.841940+0000","last_became_peered":"2024-09-16T04:09:19.841940+0000","last_unstale":"2024-09-16T04:09:43.691311+0000","last_undegraded":"2024-09-16T04:09:43.691311+0000","last_fullsized":"2024-09-16T04:09:43.691311+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:09.860925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:09.860925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:22.770968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.593889+0000","last_change":"2024-09-16T04:09:16.871123+0000","last_active":"2024-09-16T04:09:43.593889+0000","last_peered":"2024-09-16T04:09:43.593889+0000","last_clean":"2024-09-16T04:09:43.593889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:09:43.593889+0000","last_undegraded":"2024-09-16T04:09:43.593889+0000","last_fullsized":"2024-09-16T04:09:43.593889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:16.871004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:16.871004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.338451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691194+0000","last_change":"2024-09-16T04:09:19.780917+0000","last_active":"2024-09-16T04:09:43.691194+0000","last_peered":"2024-09-16T04:09:43.691194+0000","last_clean":"2024-09-16T04:09:43.691194+0000","last_became_active":"2024-09-16T04:09:19.780560+0000","last_became_peered":"2024-09-16T04:09:19.780560+0000","last_unstale":"2024-09-16T04:09:43.691194+0000","last_undegraded":"2024-09-16T04:09:43.691194+0000","last_fullsized":"2024-09-16T04:09:43.691194+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:54.852438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:54.852438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:59.574073+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"84'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937555+0000","last_change":"2024-09-16T04:09:19.779732+0000","last_active":"2024-09-16T04:09:42.937555+0000","last_peered":"2024-09-16T04:09:42.937555+0000","last_clean":"2024-09-16T04:09:42.937555+0000","last_became_active":"2024-09-16T04:09:19.776967+0000","last_became_peered":"2024-09-16T04:09:19.776967+0000","last_unstale":"2024-09-16T04:09:42.937555+0000","last_undegraded":"2024-09-16T04:09:42.937555+0000","last_fullsized":"2024-09-16T04:09:42.937555+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3","last_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_deep_scrub":"84'3","last_deep_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_clean_scrub_stamp":"2024-09-16T04:09:00.126257+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:07.593035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":224,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.358406+0000","last_change":"2024-09-16T04:09:22.362739+0000","last_active":"2024-09-16T04:09:43.358406+0000","last_peered":"2024-09-16T04:09:43.358406+0000","last_clean":"2024-09-16T04:09:43.358406+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:09:43.358406+0000","last_undegraded":"2024-09-16T04:09:43.358406+0000","last_fullsized":"2024-09-16T04:09:43.358406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:22.362652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:22.362652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.113016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691276+0000","last_change":"2024-09-16T04:08:58.876711+0000","last_active":"2024-09-16T04:09:43.691276+0000","last_peered":"2024-09-16T04:09:43.691276+0000","last_clean":"2024-09-16T04:09:43.691276+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:09:43.691276+0000","last_undegraded":"2024-09-16T04:09:43.691276+0000","last_fullsized":"2024-09-16T04:09:43.691276+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_clean_scrub_stamp":"2024-09-16T04:08:58.876583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:28.287717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":227,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937570+0000","last_change":"2024-09-16T04:08:59.147922+0000","last_active":"2024-09-16T04:09:42.937570+0000","last_peered":"2024-09-16T04:09:42.937570+0000","last_clean":"2024-09-16T04:09:42.937570+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:09:42.937570+0000","last_undegraded":"2024-09-16T04:09:42.937570+0000","last_fullsized":"2024-09-16T04:09:42.937570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-09-16T04:08:59.147797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:59.147797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:25.871032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":392,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691286+0000","last_change":"2024-09-16T04:08:55.861559+0000","last_active":"2024-09-16T04:09:43.691286+0000","last_peered":"2024-09-16T04:09:43.691286+0000","last_clean":"2024-09-16T04:09:43.691286+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:09:43.691286+0000","last_undegraded":"2024-09-16T04:09:43.691286+0000","last_fullsized":"2024-09-16T04:09:43.691286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'137","last_scrub_stamp":"2024-09-16T04:08:55.861468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:08:55.861468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:09:57.766582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378651,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50660,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321180,"statfs":{"total":96636764160,"available":96584888320,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54820,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317020,"statfs":{"total":96636764160,"available":96580628480,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378650,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50656,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378653,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58976,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312864,"statfs":{"total":96636764160,"available":96576372736,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411354,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55324,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316516,"statfs":{"total":96636764160,"available":96580112384,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411354,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50652,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321188,"statfs":{"total":96636764160,"available":96584896512,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51164,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320676,"statfs":{"total":96636764160,"available":96584372224,"internally_reserved":0,"allocated":1974272,"data_stored":1432621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":76,"seq":326417514509,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":1424,"kb_used_omap":2,"kb_used_meta":27645,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2908,"internal_metadata":28308644},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444059,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1441792,"data_stored":903693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:09:48.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:09:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:48.988 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":123,"fsid":"44381953-4509-4b15-853e-3887928dfa29","created":"2024-09-16T04:07:36.650572+0000","modified":"2024-09-16T04:09:42.931633+0000","last_up_change":"2024-09-16T04:08:51.432862+0000","last_in_change":"2024-09-16T04:09:18.766087+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:07:48.723300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:07:49.309634+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":0.75,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4506422f-7ccf-46d7-af66-62e8cace167a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6825","nonce":3110165165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6827","nonce":3110165165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6831","nonce":3110165165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3110165165},{"type":"v1","addr":"172.21.15.16:6829","nonce":3110165165}]},"public_addr":"172.21.15.16:6825/3110165165","cluster_addr":"172.21.15.16:6827/3110165165","heartbeat_back_addr":"172.21.15.16:6831/3110165165","heartbeat_front_addr":"172.21.15.16:6829/3110165165","state":["exists","up"]},{"osd":1,"uuid":"8dc0488e-efb6-4664-a799-5e14c56769b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":54,"up_from":76,"up_thru":99,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6810","nonce":4190982331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6814","nonce":4190982331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6822","nonce":4190982331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":4190982331},{"type":"v1","addr":"172.21.15.16:6818","nonce":4190982331}]},"public_addr":"172.21.15.16:6810/4190982331","cluster_addr":"172.21.15.16:6814/4190982331","heartbeat_back_addr":"172.21.15.16:6822/4190982331","heartbeat_front_addr":"172.21.15.16:6818/4190982331","state":["exists","up"]},{"osd":2,"uuid":"1686e657-c1f3-499b-8b10-67f428c5c216","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6801","nonce":2266032822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6803","nonce":2266032822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6807","nonce":2266032822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2266032822},{"type":"v1","addr":"172.21.15.16:6805","nonce":2266032822}]},"public_addr":"172.21.15.16:6801/2266032822","cluster_addr":"172.21.15.16:6803/2266032822","heartbeat_back_addr":"172.21.15.16:6807/2266032822","heartbeat_front_addr":"172.21.15.16:6805/2266032822","state":["exists","up"]},{"osd":3,"uuid":"730ad37c-1f5f-4d23-a1a7-c75320506665","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6809","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6811","nonce":3389562884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6815","nonce":3389562884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6823","nonce":3389562884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":3389562884},{"type":"v1","addr":"172.21.15.16:6819","nonce":3389562884}]},"public_addr":"172.21.15.16:6811/3389562884","cluster_addr":"172.21.15.16:6815/3389562884","heartbeat_back_addr":"172.21.15.16:6823/3389562884","heartbeat_front_addr":"172.21.15.16:6819/3389562884","state":["exists","up"]},{"osd":4,"uuid":"d82c5c44-72fe-4680-9c04-ab4e0775b929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6811","nonce":1733996396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6813","nonce":1733996396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6817","nonce":1733996396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1733996396},{"type":"v1","addr":"172.21.15.174:6815","nonce":1733996396}]},"public_addr":"172.21.15.174:6811/1733996396","cluster_addr":"172.21.15.174:6813/1733996396","heartbeat_back_addr":"172.21.15.174:6817/1733996396","heartbeat_front_addr":"172.21.15.174:6815/1733996396","state":["exists","up"]},{"osd":5,"uuid":"41502441-b0d9-460f-92c0-b8b3a915cac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6827","nonce":2956753716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6829","nonce":2956753716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6833","nonce":2956753716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2956753716},{"type":"v1","addr":"172.21.15.174:6831","nonce":2956753716}]},"public_addr":"172.21.15.174:6827/2956753716","cluster_addr":"172.21.15.174:6829/2956753716","heartbeat_back_addr":"172.21.15.174:6833/2956753716","heartbeat_front_addr":"172.21.15.174:6831/2956753716","state":["exists","up"]},{"osd":6,"uuid":"45cc7521-11db-4496-8d0c-1abc42962810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6803","nonce":2727832023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6805","nonce":2727832023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6809","nonce":2727832023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2727832023},{"type":"v1","addr":"172.21.15.174:6807","nonce":2727832023}]},"public_addr":"172.21.15.174:6803/2727832023","cluster_addr":"172.21.15.174:6805/2727832023","heartbeat_back_addr":"172.21.15.174:6809/2727832023","heartbeat_front_addr":"172.21.15.174:6807/2727832023","state":["exists","up"]},{"osd":7,"uuid":"1b5d6337-73c3-494b-91cd-fcfa7d307050","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6819","nonce":2338392542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6821","nonce":2338392542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6825","nonce":2338392542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2338392542},{"type":"v1","addr":"172.21.15.174:6823","nonce":2338392542}]},"public_addr":"172.21.15.174:6819/2338392542","cluster_addr":"172.21.15.174:6821/2338392542","heartbeat_back_addr":"172.21.15.174:6825/2338392542","heartbeat_front_addr":"172.21.15.174:6823/2338392542","state":["exists","up"]},{"osd":8,"uuid":"766efffa-3aff-4aa3-9cf8-88c5c7986d21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6801","nonce":2621487148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6803","nonce":2621487148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6807","nonce":2621487148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2621487148},{"type":"v1","addr":"172.21.15.194:6805","nonce":2621487148}]},"public_addr":"172.21.15.194:6801/2621487148","cluster_addr":"172.21.15.194:6803/2621487148","heartbeat_back_addr":"172.21.15.194:6807/2621487148","heartbeat_front_addr":"172.21.15.194:6805/2621487148","state":["exists","up"]},{"osd":9,"uuid":"755cc9aa-2f6f-4d7a-84cf-7feb74d65c48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6817","nonce":367304498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6819","nonce":367304498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6823","nonce":367304498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":367304498},{"type":"v1","addr":"172.21.15.194:6821","nonce":367304498}]},"public_addr":"172.21.15.194:6817/367304498","cluster_addr":"172.21.15.194:6819/367304498","heartbeat_back_addr":"172.21.15.194:6823/367304498","heartbeat_front_addr":"172.21.15.194:6821/367304498","state":["exists","up"]},{"osd":10,"uuid":"76a2c6a8-55e4-4cbd-ba0d-bb7a0ed6d377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6825","nonce":3349503344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6827","nonce":3349503344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6831","nonce":3349503344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3349503344},{"type":"v1","addr":"172.21.15.194:6829","nonce":3349503344}]},"public_addr":"172.21.15.194:6825/3349503344","cluster_addr":"172.21.15.194:6827/3349503344","heartbeat_back_addr":"172.21.15.194:6831/3349503344","heartbeat_front_addr":"172.21.15.194:6829/3349503344","state":["exists","up"]},{"osd":11,"uuid":"f71641eb-1331-4b39-9c0e-bf98e91ed295","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6809","nonce":2531896090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6811","nonce":2531896090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6815","nonce":2531896090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2531896090},{"type":"v1","addr":"172.21.15.194:6813","nonce":2531896090}]},"public_addr":"172.21.15.194:6809/2531896090","cluster_addr":"172.21.15.194:6811/2531896090","heartbeat_back_addr":"172.21.15.194:6815/2531896090","heartbeat_front_addr":"172.21.15.194:6813/2531896090","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:01.680673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:58.907614+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:47.992191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:48.214554+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:15.656396+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:59.631647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:58.489685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:08.502081+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:48.771857+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:06.156463+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:08:56.419990+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:09:00.745572+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lrcprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","l":"3","m":"2","name":"lrcprofile","plugin":"lrc"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":22,"length":1},{"begin":27,"length":1},{"begin":31,"length":1},{"begin":33,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:09:50.000 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T04:09:50.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:50.149 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:50.149 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:50.149 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:50.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T04:09:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T04:09:50.468 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T04:09:50.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:50.603 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:50.603 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:50.603 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:50.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T04:09:50.909 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T04:09:50.922 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T04:09:50.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:51.058 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:51.058 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:51.059 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:51.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T04:09:51.364 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T04:09:51.378 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T04:09:51.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:51.515 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:51.515 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:51.515 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:51.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T04:09:51.822 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T04:09:51.835 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T04:09:51.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:51.972 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:51.972 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:51.972 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:51.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T04:09:52.277 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T04:09:52.290 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T04:09:52.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:52.428 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:52.428 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:52.428 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:52.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T04:09:52.731 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T04:09:52.744 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T04:09:52.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:52.882 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:52.882 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:52.882 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:52.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T04:09:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T04:09:53.200 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T04:09:53.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:53.346 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T04:09:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T04:09:53.655 INFO:tasks.ceph:Scrubbing osd.8 2024-09-16T04:09:53.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:53.793 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:53.793 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:53.793 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:53.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-16T04:09:54.103 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 8 to deep-scrub 2024-09-16T04:09:54.116 INFO:tasks.ceph:Scrubbing osd.9 2024-09-16T04:09:54.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:54.252 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:54.253 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:54.253 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:54.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-16T04:09:54.560 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 9 to deep-scrub 2024-09-16T04:09:54.574 INFO:tasks.ceph:Scrubbing osd.10 2024-09-16T04:09:54.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:54.710 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:54.710 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:54.710 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:54.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-16T04:09:55.019 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 10 to deep-scrub 2024-09-16T04:09:55.032 INFO:tasks.ceph:Scrubbing osd.11 2024-09-16T04:09:55.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:09:55.168 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-16T04:09:55.168 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:09:55.168 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-16T04:09:55.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-16T04:09:55.477 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 11 to deep-scrub 2024-09-16T04:09:55.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:09:55.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:09:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:09:55.791 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-16T04:09:54.694314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":540152,"kb_used_data":18088,"kb_used_omap":19,"kb_used_meta":521772,"kb_avail":1131921928,"statfs":{"total":1159641169920,"available":1159088054272,"internally_reserved":0,"allocated":18522112,"data_stored":11994802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20453,"internal_metadata":534294555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.757690"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":225,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.936638+0000","last_change":"2024-09-16T04:09:08.905295+0000","last_active":"2024-09-16T04:09:42.936638+0000","last_peered":"2024-09-16T04:09:42.936638+0000","last_clean":"2024-09-16T04:09:42.936638+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:09:42.936638+0000","last_undegraded":"2024-09-16T04:09:42.936638+0000","last_fullsized":"2024-09-16T04:09:42.936638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:08.905173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:08.905173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:32.060373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":54,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.691349+0000","last_change":"2024-09-16T04:09:19.842195+0000","last_active":"2024-09-16T04:09:43.691349+0000","last_peered":"2024-09-16T04:09:43.691349+0000","last_clean":"2024-09-16T04:09:43.691349+0000","last_became_active":"2024-09-16T04:09:19.841940+0000","last_became_peered":"2024-09-16T04:09:19.841940+0000","last_unstale":"2024-09-16T04:09:43.691349+0000","last_undegraded":"2024-09-16T04:09:43.691349+0000","last_fullsized":"2024-09-16T04:09:43.691349+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:09.860925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:09.860925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":222,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.593889+0000","last_change":"2024-09-16T04:09:16.871123+0000","last_active":"2024-09-16T04:09:43.593889+0000","last_peered":"2024-09-16T04:09:43.593889+0000","last_clean":"2024-09-16T04:09:43.593889+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:09:43.593889+0000","last_undegraded":"2024-09-16T04:09:43.593889+0000","last_fullsized":"2024-09-16T04:09:43.593889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:16.871004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:16.871004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.338451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":62,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:50.942319+0000","last_change":"2024-09-16T04:09:50.942356+0000","last_active":"2024-09-16T04:09:50.942319+0000","last_peered":"2024-09-16T04:09:50.942319+0000","last_clean":"2024-09-16T04:09:50.942319+0000","last_became_active":"2024-09-16T04:09:19.780560+0000","last_became_peered":"2024-09-16T04:09:19.780560+0000","last_unstale":"2024-09-16T04:09:50.942319+0000","last_undegraded":"2024-09-16T04:09:50.942319+0000","last_fullsized":"2024-09-16T04:09:50.942319+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:50.942218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:50.942218+0000","last_clean_scrub_stamp":"2024-09-16T04:09:50.942218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:04.356198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"84'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937555+0000","last_change":"2024-09-16T04:09:19.779732+0000","last_active":"2024-09-16T04:09:42.937555+0000","last_peered":"2024-09-16T04:09:42.937555+0000","last_clean":"2024-09-16T04:09:42.937555+0000","last_became_active":"2024-09-16T04:09:19.776967+0000","last_became_peered":"2024-09-16T04:09:19.776967+0000","last_unstale":"2024-09-16T04:09:42.937555+0000","last_undegraded":"2024-09-16T04:09:42.937555+0000","last_fullsized":"2024-09-16T04:09:42.937555+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3","last_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_deep_scrub":"84'3","last_deep_scrub_stamp":"2024-09-16T04:09:00.126257+0000","last_clean_scrub_stamp":"2024-09-16T04:09:00.126257+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:07.593035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":224,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:43.358406+0000","last_change":"2024-09-16T04:09:22.362739+0000","last_active":"2024-09-16T04:09:43.358406+0000","last_peered":"2024-09-16T04:09:43.358406+0000","last_clean":"2024-09-16T04:09:43.358406+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:09:43.358406+0000","last_undegraded":"2024-09-16T04:09:43.358406+0000","last_fullsized":"2024-09-16T04:09:43.358406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:22.362652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:09:22.362652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:26.113016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":224,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:53.641463+0000","last_change":"2024-09-16T04:08:58.876711+0000","last_active":"2024-09-16T04:09:53.641463+0000","last_peered":"2024-09-16T04:09:53.641463+0000","last_clean":"2024-09-16T04:09:53.641463+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:09:53.641463+0000","last_undegraded":"2024-09-16T04:09:53.641463+0000","last_fullsized":"2024-09-16T04:09:53.641463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:08:58.876583+0000","last_clean_scrub_stamp":"2024-09-16T04:08:58.876583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'1","reported_seq":227,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:42.937570+0000","last_change":"2024-09-16T04:08:59.147922+0000","last_active":"2024-09-16T04:09:42.937570+0000","last_peered":"2024-09-16T04:09:42.937570+0000","last_clean":"2024-09-16T04:09:42.937570+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:09:42.937570+0000","last_undegraded":"2024-09-16T04:09:42.937570+0000","last_fullsized":"2024-09-16T04:09:42.937570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-09-16T04:08:59.147797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:49.925832+0000","last_clean_scrub_stamp":"2024-09-16T04:08:59.147797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:25.871032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":394,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-16T04:09:53.641434+0000","last_change":"2024-09-16T04:08:55.861559+0000","last_active":"2024-09-16T04:09:53.641434+0000","last_peered":"2024-09-16T04:09:53.641434+0000","last_clean":"2024-09-16T04:09:53.641434+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:09:53.641434+0000","last_undegraded":"2024-09-16T04:09:53.641434+0000","last_fullsized":"2024-09-16T04:09:53.641434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'137","last_scrub_stamp":"2024-09-16T04:08:55.861468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:07:48.919317+0000","last_clean_scrub_stamp":"2024-09-16T04:08:55.861468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378652,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50660,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321180,"statfs":{"total":96636764160,"available":96584888320,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54820,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317020,"statfs":{"total":96636764160,"available":96580628480,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50656,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378654,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58976,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312864,"statfs":{"total":96636764160,"available":96576372736,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55340,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316500,"statfs":{"total":96636764160,"available":96580096000,"internally_reserved":0,"allocated":1986560,"data_stored":1440337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411356,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50656,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51180,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320660,"statfs":{"total":96636764160,"available":96584355840,"internally_reserved":0,"allocated":1986560,"data_stored":1440337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":76,"seq":326417514510,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":1424,"kb_used_omap":2,"kb_used_meta":27645,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":1458176,"data_stored":911428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2908,"internal_metadata":28308644},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444061,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36776,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335064,"statfs":{"total":96636764160,"available":96599105536,"internally_reserved":0,"allocated":1454080,"data_stored":911409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:09:55.796 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T04:09:08.905173+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=8, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=48, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:09:55.796 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T04:09:09.860925+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=48, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:09:55.796 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T04:09:16.871004+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=48, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:09:55.796 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T04:09:00.126257+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=48, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:09:55.796 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T04:09:22.362652+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=48, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:09:55.796 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T04:08:58.876583+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=8, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=48, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:09:55.797 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T04:08:59.147797+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=8, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=48, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:09:55.797 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T04:08:55.861468+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=8, tm_sec=55, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=48, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:09:55.797 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T04:10:15.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:10:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-16T04:10:16.107 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-16T04:10:16.119 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-16T04:10:15.759638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":183,"num_write_kb":2940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1132462080,"kb_used":540908,"kb_used_data":18808,"kb_used_omap":19,"kb_used_meta":521772,"kb_avail":1131921172,"statfs":{"total":1159641169920,"available":1159087280128,"internally_reserved":0,"allocated":19259392,"data_stored":12460282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20453,"internal_metadata":534294555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.991067"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":245,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T04:10:07.962097+0000","last_change":"2024-09-16T04:09:53.728615+0000","last_active":"2024-09-16T04:10:07.962097+0000","last_peered":"2024-09-16T04:10:07.962097+0000","last_clean":"2024-09-16T04:10:07.962097+0000","last_became_active":"2024-09-16T04:07:51.226905+0000","last_became_peered":"2024-09-16T04:07:51.226905+0000","last_unstale":"2024-09-16T04:10:07.962097+0000","last_undegraded":"2024-09-16T04:10:07.962097+0000","last_fullsized":"2024-09-16T04:10:07.962097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:53.728494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:53.728494+0000","last_clean_scrub_stamp":"2024-09-16T04:09:53.728494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:56.758703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037006700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":72,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T04:10:07.942201+0000","last_change":"2024-09-16T04:09:51.899330+0000","last_active":"2024-09-16T04:10:07.942201+0000","last_peered":"2024-09-16T04:10:07.942201+0000","last_clean":"2024-09-16T04:10:07.942201+0000","last_became_active":"2024-09-16T04:09:19.841940+0000","last_became_peered":"2024-09-16T04:09:19.841940+0000","last_unstale":"2024-09-16T04:10:07.942201+0000","last_undegraded":"2024-09-16T04:10:07.942201+0000","last_fullsized":"2024-09-16T04:10:07.942201+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:51.899238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:51.899238+0000","last_clean_scrub_stamp":"2024-09-16T04:09:51.899238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:11.294971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":242,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T04:10:07.962477+0000","last_change":"2024-09-16T04:09:54.610043+0000","last_active":"2024-09-16T04:10:07.962477+0000","last_peered":"2024-09-16T04:10:07.962477+0000","last_clean":"2024-09-16T04:10:07.962477+0000","last_became_active":"2024-09-16T04:07:50.936451+0000","last_became_peered":"2024-09-16T04:07:50.936451+0000","last_unstale":"2024-09-16T04:10:07.962477+0000","last_undegraded":"2024-09-16T04:10:07.962477+0000","last_fullsized":"2024-09-16T04:10:07.962477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:54.609898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:54.609898+0000","last_clean_scrub_stamp":"2024-09-16T04:09:54.609898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:06.836239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043345599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":72,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T04:10:07.942130+0000","last_change":"2024-09-16T04:09:50.942356+0000","last_active":"2024-09-16T04:10:07.942130+0000","last_peered":"2024-09-16T04:10:07.942130+0000","last_clean":"2024-09-16T04:10:07.942130+0000","last_became_active":"2024-09-16T04:09:19.780560+0000","last_became_peered":"2024-09-16T04:09:19.780560+0000","last_unstale":"2024-09-16T04:10:07.942130+0000","last_undegraded":"2024-09-16T04:10:07.942130+0000","last_fullsized":"2024-09-16T04:10:07.942130+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:50.942218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:50.942218+0000","last_clean_scrub_stamp":"2024-09-16T04:09:50.942218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:04.356198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"84'3","reported_seq":266,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T04:10:07.770825+0000","last_change":"2024-09-16T04:09:54.477141+0000","last_active":"2024-09-16T04:10:07.770825+0000","last_peered":"2024-09-16T04:10:07.770825+0000","last_clean":"2024-09-16T04:10:07.770825+0000","last_became_active":"2024-09-16T04:09:19.776967+0000","last_became_peered":"2024-09-16T04:09:19.776967+0000","last_unstale":"2024-09-16T04:10:07.770825+0000","last_undegraded":"2024-09-16T04:10:07.770825+0000","last_fullsized":"2024-09-16T04:10:07.770825+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'3","last_scrub_stamp":"2024-09-16T04:09:54.477008+0000","last_deep_scrub":"84'3","last_deep_scrub_stamp":"2024-09-16T04:09:54.477008+0000","last_clean_scrub_stamp":"2024-09-16T04:09:54.477008+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:15.937765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070524300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":244,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T04:10:07.962549+0000","last_change":"2024-09-16T04:09:52.359243+0000","last_active":"2024-09-16T04:10:07.962549+0000","last_peered":"2024-09-16T04:10:07.962549+0000","last_clean":"2024-09-16T04:10:07.962549+0000","last_became_active":"2024-09-16T04:07:50.944437+0000","last_became_peered":"2024-09-16T04:07:50.944437+0000","last_unstale":"2024-09-16T04:10:07.962549+0000","last_undegraded":"2024-09-16T04:10:07.962549+0000","last_fullsized":"2024-09-16T04:10:07.962549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:52.359099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:52.359099+0000","last_clean_scrub_stamp":"2024-09-16T04:09:52.359099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:08.150937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049697299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":242,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T04:10:07.962329+0000","last_change":"2024-09-16T04:09:55.035201+0000","last_active":"2024-09-16T04:10:07.962329+0000","last_peered":"2024-09-16T04:10:07.962329+0000","last_clean":"2024-09-16T04:10:07.962329+0000","last_became_active":"2024-09-16T04:07:51.226929+0000","last_became_peered":"2024-09-16T04:07:51.226929+0000","last_unstale":"2024-09-16T04:10:07.962329+0000","last_undegraded":"2024-09-16T04:10:07.962329+0000","last_fullsized":"2024-09-16T04:10:07.962329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:55.035091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:55.035091+0000","last_clean_scrub_stamp":"2024-09-16T04:09:55.035091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:18.991709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034732399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"123'2","reported_seq":248,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T04:10:07.770825+0000","last_change":"2024-09-16T04:09:53.447998+0000","last_active":"2024-09-16T04:10:07.770825+0000","last_peered":"2024-09-16T04:10:07.770825+0000","last_clean":"2024-09-16T04:10:07.770825+0000","last_became_active":"2024-09-16T04:07:50.935270+0000","last_became_peered":"2024-09-16T04:07:50.935270+0000","last_unstale":"2024-09-16T04:10:07.770825+0000","last_undegraded":"2024-09-16T04:10:07.770825+0000","last_fullsized":"2024-09-16T04:10:07.770825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'2","last_scrub_stamp":"2024-09-16T04:09:53.447873+0000","last_deep_scrub":"123'2","last_deep_scrub_stamp":"2024-09-16T04:09:53.447873+0000","last_clean_scrub_stamp":"2024-09-16T04:09:53.447873+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:56.296223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00036370800000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"123'139","reported_seq":414,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T04:10:07.962340+0000","last_change":"2024-09-16T04:09:54.033055+0000","last_active":"2024-09-16T04:10:07.962340+0000","last_peered":"2024-09-16T04:10:07.962340+0000","last_clean":"2024-09-16T04:10:07.962340+0000","last_became_active":"2024-09-16T04:07:50.119023+0000","last_became_peered":"2024-09-16T04:07:50.119023+0000","last_unstale":"2024-09-16T04:10:07.962340+0000","last_undegraded":"2024-09-16T04:10:07.962340+0000","last_fullsized":"2024-09-16T04:10:07.962340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'139","last_scrub_stamp":"2024-09-16T04:09:54.032925+0000","last_deep_scrub":"123'139","last_deep_scrub_stamp":"2024-09-16T04:09:54.032925+0000","last_clean_scrub_stamp":"2024-09-16T04:09:54.032925+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:11:02.728384+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":19,"seq":81604378657,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50724,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321116,"statfs":{"total":96636764160,"available":96584822784,"internally_reserved":0,"allocated":1515520,"data_stored":950199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":19,"seq":81604378656,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54884,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316956,"statfs":{"total":96636764160,"available":96580562944,"internally_reserved":0,"allocated":1519616,"data_stored":950218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378656,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50720,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321120,"statfs":{"total":96636764160,"available":96584826880,"internally_reserved":0,"allocated":1515520,"data_stored":950199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":19,"seq":81604378659,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59040,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312800,"statfs":{"total":96636764160,"available":96576307200,"internally_reserved":0,"allocated":1515520,"data_stored":950199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378657,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36836,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335004,"statfs":{"total":96636764160,"available":96599044096,"internally_reserved":0,"allocated":1515520,"data_stored":950199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":18,"seq":77309411360,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55404,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316436,"statfs":{"total":96636764160,"available":96580030464,"internally_reserved":0,"allocated":2048000,"data_stored":1479127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":18,"seq":77309411362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1515520,"data_stored":950199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":18,"seq":77309411360,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50720,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321120,"statfs":{"total":96636764160,"available":96584826880,"internally_reserved":0,"allocated":1515520,"data_stored":950199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":18,"seq":77309411360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1515520,"data_stored":950199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":2048000,"data_stored":1479127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":76,"seq":326417514514,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29160,"kb_used_data":1484,"kb_used_omap":2,"kb_used_meta":27645,"kb_avail":94342680,"statfs":{"total":96636764160,"available":96606904320,"internally_reserved":0,"allocated":1519616,"data_stored":950218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2908,"internal_metadata":28308644},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":17,"seq":73014444065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36836,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335004,"statfs":{"total":96636764160,"available":96599044096,"internally_reserved":0,"allocated":1515520,"data_stored":950199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:10:16.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T04:10:16.433 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T04:10:16.433 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T04:10:16.433 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T04:10:16.433 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:16.478 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T04:10:16.479 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:10:16.479 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:16.513 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:10:16.513 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T04:10:16.513 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:16.588 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T04:10:16.589 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T04:10:16.589 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:16.660 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T04:10:16.661 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T04:10:16.661 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:16.709 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T04:10:16.709 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T04:10:16.710 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:16.744 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T04:10:16.744 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T04:10:16.744 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:16.818 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T04:10:16.819 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T04:10:16.819 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:16.898 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T04:10:16.899 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-16T04:10:16.899 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:16.930 INFO:tasks.ceph.osd.8:Stopped 2024-09-16T04:10:16.930 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-16T04:10:16.930 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:17.001 INFO:tasks.ceph.osd.9:Stopped 2024-09-16T04:10:17.001 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-16T04:10:17.001 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:17.071 INFO:tasks.ceph.osd.10:Stopped 2024-09-16T04:10:17.071 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-16T04:10:17.072 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:17.148 INFO:tasks.ceph.osd.11:Stopped 2024-09-16T04:10:17.149 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T04:10:17.149 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T04:10:17.149 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:17.180 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T04:10:17.180 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T04:10:17.181 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:10:17.181 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:17.193 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:10:17.193 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:10:17.193 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:17.235 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:10:17.235 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:10:17.235 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:17.248 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:10:17.248 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T04:10:17.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-16T04:10:17.288 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:10:17.288 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T04:10:17.538 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:10:17.538 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T04:10:17.610 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:10:17.611 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T04:10:17.670 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:10:17.670 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T04:10:17.729 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:10:17.729 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T04:10:17.898 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:10:17.898 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T04:10:17.985 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:10:17.986 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T04:10:18.043 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:10:18.044 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T04:10:18.117 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:10:18.118 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-16T04:10:18.268 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:10:18.268 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-16T04:10:18.357 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:10:18.358 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-16T04:10:18.413 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:10:18.413 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-16T04:10:18.462 INFO:tasks.ceph:Archiving mon data... 2024-09-16T04:10:18.464 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/data/mon.a.tgz 2024-09-16T04:10:18.464 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-16T04:10:18.483 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.eY46nituYd 2024-09-16T04:10:18.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.eY46nituYd 2024-09-16T04:10:18.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.eY46nituYd 2024-09-16T04:10:18.720 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.eY46nituYd is 314KB 2024-09-16T04:10:18.817 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.eY46nituYd 2024-09-16T04:10:18.833 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/data/mon.b.tgz 2024-09-16T04:10:18.834 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-09-16T04:10:18.853 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.QU7V1iiwdd 2024-09-16T04:10:18.853 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.QU7V1iiwdd 2024-09-16T04:10:19.024 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.QU7V1iiwdd 2024-09-16T04:10:19.108 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.QU7V1iiwdd is 434KB 2024-09-16T04:10:19.209 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.QU7V1iiwdd 2024-09-16T04:10:19.224 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/data/mon.c.tgz 2024-09-16T04:10:19.224 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-09-16T04:10:19.243 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.49BA2Phe9O 2024-09-16T04:10:19.244 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.49BA2Phe9O 2024-09-16T04:10:19.418 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.49BA2Phe9O 2024-09-16T04:10:19.501 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.49BA2Phe9O is 434KB 2024-09-16T04:10:19.599 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.49BA2Phe9O 2024-09-16T04:10:19.614 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T04:10:19.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:10:19.617 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:10:19.620 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:10:19.702 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:10:19.719 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:10:19.735 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:10:19.735 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T04:10:19.736 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/remote/smithi016/crash 2024-09-16T04:10:19.737 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:10:19.771 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/remote/smithi174/crash 2024-09-16T04:10:19.772 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:10:19.798 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/remote/smithi194/crash 2024-09-16T04:10:19.798 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:10:19.822 INFO:tasks.ceph:Compressing logs... 2024-09-16T04:10:19.823 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:10:19.826 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:10:19.842 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:10:19.850 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T04:10:19.868 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T04:10:19.868 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T04:10:19.868 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:10:19.868 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T04:10:19.868 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T04:10:19.868 INFO:teuthology.orchestra.run.smithi174.stderr:gzip/var/log/ceph/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:10:19.869 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T04:10:19.869 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T04:10:19.869 INFO:teuthology.orchestra.run.smithi174.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:10:19.869 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-mon.b.log: 85.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:10:19.912 INFO:teuthology.orchestra.run.smithi174.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T04:10:19.914 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-16T04:10:19.914 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32920.log 2024-09-16T04:10:19.914 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33120.log 2024-09-16T04:10:19.914 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.32920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32920.log.gz 2024-09-16T04:10:19.914 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33000.log 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.33120.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33120.log.gz 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.33000.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.33080.log -- replaced with /var/log/ceph/ceph-client.admin.33000.log.gz 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33040.log 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.33080.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33080.log.gz 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.33040.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33040.log.gz -5 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32760.log 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.32760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32760.log.gz 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32840.log 2024-09-16T04:10:19.915 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.32840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33200.log 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32840.log.gz 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32880.log 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.33200.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33200.log.gz 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr: /var/log/ceph/ceph-osd.8.log 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.32880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32880.log.gz 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32960.log 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33160.log 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr: 85.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.32960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32960.log.gz 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32800.log 2024-09-16T04:10:19.916 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.33160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33160.log.gz 2024-09-16T04:10:19.917 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.32800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32800.log.gz 2024-09-16T04:10:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52528.log 2024-09-16T04:10:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39838.log 2024-09-16T04:10:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53596.log 2024-09-16T04:10:19.958 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52528.log.gz 2024-09-16T04:10:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49286.log 2024-09-16T04:10:19.958 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39838.log.gz 2024-09-16T04:10:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54200.log 2024-09-16T04:10:19.958 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53596.log.gz 2024-09-16T04:10:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39343.log 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49286.log.gz 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52100.log 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54200.log.gz 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39343.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39343.log.gz /var/log/ceph/ceph.tmp-client.admin.27142.log 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42109.log 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52100.log.gz 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.tmp-client.admin.27142.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.40560.log 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.27142.log.gz 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38188.log 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42109.log.gz 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53948.log 2024-09-16T04:10:19.959 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38188.log.gz 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53626.log 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42541.log 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53948.log.gz 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51171.log 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53626.log.gz 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42541.log.gz 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54746.log 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51171.log.gz 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41182.log 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-09-16T04:10:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52530.log 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54746.log.gz 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38187.log 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41182.log.gz 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52530.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.52530.log.gz --verbose 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/ceph-client.admin.38172.log 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38903.log 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38187.log.gz 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32863.log 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr: 26.0% -- replaced with /var/log/ceph/ceph-client.admin.38172.log.gz 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40495.log 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38903.log.gz 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52971.log 0.0% 2024-09-16T04:10:19.961 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32863.log.gz 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52511.log 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40495.log.gz 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53027.log 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52971.log.gz 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52497.log 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52511.log.gz 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53027.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.51722.log 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53027.log.gz 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38575.log 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52497.log.gz 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54830.log 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51722.log.gz 2024-09-16T04:10:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53864.log 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38575.log.gz 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54830.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54830.log.gz 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39914.log 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53864.log.gz 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55124.log 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39914.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39914.log.gz 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41224.log 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38987.log 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55124.log.gz 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39197.log 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41224.log.gz 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48279.log 2024-09-16T04:10:19.963 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38987.log.gz 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48831.log 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39197.log.gz 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48279.log.gz 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53388.log 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50835.log 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48831.log.gz 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42174.log 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53388.log.gz 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49245.log 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50835.log.gz 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52508.log 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42174.log.gz 2024-09-16T04:10:19.964 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42857.log 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49245.log.gz 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55040.log 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52508.log.gz 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41720.log 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42857.log.gz 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52507.log 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55040.log.gz 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41720.log.gz 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50214.log 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50815.log 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52507.log.gz 2024-09-16T04:10:19.965 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54872.log 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50214.log.gz 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40311.log 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50815.log.gz 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38197.log 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54872.log.gz 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53496.log 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40311.log.gz 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51932.log 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38197.log.gz 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50671.log 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53496.log.gz 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40331.log 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51932.log.gz 2024-09-16T04:10:19.966 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54116.log 2024-09-16T04:10:19.967 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50671.log.gz 2024-09-16T04:10:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40331.log.gz 2024-09-16T04:10:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39113.log 2024-09-16T04:10:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50813.log 2024-09-16T04:10:19.967 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54116.log.gz 2024-09-16T04:10:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54662.log 2024-09-16T04:10:19.967 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39113.log.gz 2024-09-16T04:10:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42754.log 2024-09-16T04:10:19.967 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50813.log.gz 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54032.log 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54662.log.gz 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40229.log 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42754.log.gz 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52525.log 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54032.log.gz 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40229.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40229.log.gz 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39465.log 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54998.log 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52525.log.gz 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50280.log 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39465.log.gz 2024-09-16T04:10:19.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50321.log 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54998.log.gz 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50280.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.44942.log 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50280.log.gz 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53358.log 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.50321.log.gz 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/objectstore_tool.44942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55082.log 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53358.log.gz 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52994.log 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54494.log 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55082.log.gz 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39155.log 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52994.log.gz 2024-09-16T04:10:19.969 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54494.log.gz 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39695.log 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39155.log.gz 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39695.log.gz 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37679.log 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53032.log 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.37679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54368.log 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37679.log.gz 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53032.log.gzgzip 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52521.log 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54242.log 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54368.log.gz 2024-09-16T04:10:19.970 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39071.log 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52521.log.gz 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54242.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.54242.log.gz 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53495.log 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52058.log 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39071.log.gz 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53495.log.gz 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50487.log 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44966.log 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52058.log.gz 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50487.log.gz 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47310.log 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38648.log 2024-09-16T04:10:19.971 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44966.log.gz 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51806.log 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47310.log.gz 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38648.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.38648.log.gz 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr: /var/log/ceph/ceph-client.admin.42215.log 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51386.log 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51806.log.gz 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42215.log.gz 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52184.log 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53033.log 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51386.log.gz 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47967.log 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52184.log.gz 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54956.log 2024-09-16T04:10:19.972 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53033.log.gz 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53822.log 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47967.log.gz 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54956.log.gz 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53022.log 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54536.log 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53822.log.gz 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53022.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.53022.log.gz -5 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50631.log 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54788.log 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54536.log.gz 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50631.log.gz 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51638.log 2024-09-16T04:10:19.973 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51344.log 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54788.log.gz 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51638.log.gz 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42538.log 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38184.log 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51344.log.gz 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52268.log 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42538.log.gz 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54620.log 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38184.log.gz 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52268.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52268.log.gz 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/ceph-client.admin.41633.log 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54620.log.gz 2024-09-16T04:10:19.974 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38170.log 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42648.log 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41633.log.gz 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39239.log 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr: 26.8% -- replaced with /var/log/ceph/ceph-client.admin.38170.log.gz 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54410.log 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42648.log.gz 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38571.log 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39239.log.gz 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54326.log 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54410.log.gz 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38571.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.38571.log.gz /var/log/ceph/ceph-client.admin.33198.log 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51218.log 2024-09-16T04:10:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54326.log.gz 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54452.log 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33198.log.gz 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51218.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51218.log.gz -5 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40062.log 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50985.log 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54452.log.gz 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40062.log.gz 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51554.log 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37895.log 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50985.log.gz 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51554.log.gz 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44617.log 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr: 93.5% -- replaced with /var/log/ceph/objectstore_tool.44942.log.gz 2024-09-16T04:10:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43489.log 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37895.log.gz 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44617.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.44617.log.gz /var/log/ceph/bluestore_tool.47740.log 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51087.log 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43489.log.gz 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/bluestore_tool.47740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51764.log 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38646.log 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51087.log.gz 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54074.log 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51764.log.gz 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52512.log 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38646.log.gz 2024-09-16T04:10:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54074.log.gz 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38174.log 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52514.log 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52512.log.gz 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38174.log.gz 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39990.log 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51129.log 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52514.log.gz 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.39990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48938.log 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr: 72.9% -- replaced with /var/log/ceph/ceph-client.0.39990.log.gz 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38649.log 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51129.log.gz 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51596.log 2024-09-16T04:10:19.978 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48938.log.gz 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38649.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr: 91.8% -- replaced with /var/log/ceph/bluestore_tool.47740.log.gz 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38649.log.gz 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51680.log 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51596.log.gz 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49537.log 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51680.log.gz 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41140.log 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49848.log 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49537.log.gz 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52142.log 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41140.log.gz 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49848.log.gz 2024-09-16T04:10:19.979 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51302.log 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37937.log 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52142.log.gz 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51302.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.46797.log 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51302.log.gz 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37937.log.gz 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51512.log 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/bluestore_tool.46797.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51428.log 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51512.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51512.log.gz 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37737.log 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51428.log.gz 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41907.log 2024-09-16T04:10:19.980 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37737.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37737.log.gz 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr: /var/log/ceph/ceph-client.admin.38461.log 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41907.log.gz 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55207.log 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38461.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38461.log.gz 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53009.log 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55207.log.gz 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53498.log 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53009.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53009.log.gz 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50093.log 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53498.log.gz 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40082.log 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50093.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50093.log.gz 2024-09-16T04:10:19.981 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54158.log 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40082.log.gz 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43900.log 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54158.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54158.log.gz 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53016.log 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43900.log.gz 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51890.log 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53016.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53016.log.gz 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49806.log 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51890.log: 93.6% -- replaced with /var/log/ceph/bluestore_tool.46797.log.gz 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51890.log.gz 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53015.log 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39463.log 2024-09-16T04:10:19.982 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49806.log.gz 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53015.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53015.log.gz 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr: /var/log/ceph/ceph-client.admin.52226.log 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39463.log.gz 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38191.log 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48431.log 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52226.log.gz 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.38191.log: -5 --verbose -- /var/log/ceph/ceph.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38191.log.gz 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48431.log.gz 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54578.log 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53906.log 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49477.log 2024-09-16T04:10:19.983 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54578.log.gz 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51260.log 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53906.log.gz 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.45766.log 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49477.log: 85.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53318.log 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49477.log.gz 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51260.log.gz 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/objectstore_tool.45766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55167.log 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53318.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.43033.log 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51470.log 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55167.log.gz 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53318.log.gz 2024-09-16T04:10:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/objectstore_tool.43033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53357.log 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38901.log 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51470.log.gz 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53357.log.gz 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38628.log 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40602.log 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38901.log.gz 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38628.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39921.log 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41761.log 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40602.log.gz 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39921.log.gz 2024-09-16T04:10:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38557.log 2024-09-16T04:10:19.986 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47881.log 2024-09-16T04:10:19.986 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41761.log.gz 2024-09-16T04:10:19.986 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38557.log.gz 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42709.log 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40912.log 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47881.log.gz 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42709.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42709.log.gzgzip 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47248.log 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40912.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.46733.log 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40912.log.gz 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47248.log.gz 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52016.log 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/bluestore_tool.46733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49352.log 2024-09-16T04:10:19.987 INFO:teuthology.orchestra.run.smithi016.stderr: 85.3% -- replaced with /var/log/ceph/bluestore_tool.46733.log.gz 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52016.log.gz 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39029.log 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38199.log 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49352.log.gz 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39029.log.gz 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38193.log 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38199.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.43918.log 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38199.log.gz 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38193.log.gz 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41098.log 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/objectstore_tool.43918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40990.log 2024-09-16T04:10:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41098.log.gz 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37762.log 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40990.log.gz 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52526.log 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37762.log: 93.3% -- replaced with /var/log/ceph/objectstore_tool.45766.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37762.log.gz 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.47286.log 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50507.log 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52526.log.gz 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/bluestore_tool.47286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53034.log 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50507.log.gz 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40927.log 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53034.log.gz 2024-09-16T04:10:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38611.log 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40927.log.gz 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41677.log 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38611.log.gz 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49741.log 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41677.log.gz 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48873.log 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49741.log.gz 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43057.log 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48873.log.gz 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53497.log 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43057.log.gz 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52872.log 2024-09-16T04:10:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53497.log.gz 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38194.log 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/objectstore_tool.43918.log.gz 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52872.log.gz 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41030.log 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50814.log 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38194.log: /var/log/ceph/ceph-client.admin.41030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38194.log.gz 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41030.log.gz 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38177.log 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45797.log 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50814.log.gz 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38177.log.gz 93.3% -- replaced with /var/log/ceph/bluestore_tool.47286.log.gz 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54914.log 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33092.log 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45797.log: /var/log/ceph/ceph-client.admin.54914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39466.log 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45797.log.gz 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54914.log.gz 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33092.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.41571.log 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33092.log.gz 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53604.log 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39466.log: /var/log/ceph/ceph-client.admin.41571.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39466.log.gz 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41571.log.gz 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53604.log.gz 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53579.log 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52924.log 2024-09-16T04:10:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.53579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53990.log 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53579.log.gz 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52924.log.gz 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48979.log 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54284.log 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53990.log.gz 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48979.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51974.log 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37845.log 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54284.log.gz 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51974.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54704.log 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48213.log 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37845.log.gz 2024-09-16T04:10:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.54704.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52516.log 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54704.log.gz 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39281.log 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48213.log: /var/log/ceph/ceph-client.admin.52516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48213.log.gz 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52516.log.gz 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38559.log 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51848.log 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39281.log.gz 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38559.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46390.log 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51848.log.gz 2024-09-16T04:10:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46390.log.gz 2024-09-16T04:10:19.995 INFO:teuthology.orchestra.run.smithi016.stderr: 94.2% -- replaced with /var/log/ceph/objectstore_tool.43033.log.gz 2024-09-16T04:10:20.004 INFO:teuthology.orchestra.run.smithi174.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T04:10:20.009 INFO:teuthology.orchestra.run.smithi194.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T04:10:20.102 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T04:10:20.342 INFO:teuthology.orchestra.run.smithi016.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T04:10:20.462 INFO:teuthology.orchestra.run.smithi194.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-16T04:10:20.468 INFO:teuthology.orchestra.run.smithi194.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-16T04:10:20.532 INFO:teuthology.orchestra.run.smithi194.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-16T04:10:20.573 INFO:teuthology.orchestra.run.smithi174.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T04:10:20.795 INFO:teuthology.orchestra.run.smithi016.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T04:10:20.928 INFO:teuthology.orchestra.run.smithi174.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T04:10:20.940 INFO:teuthology.orchestra.run.smithi174.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T04:10:21.018 INFO:teuthology.orchestra.run.smithi194.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-16T04:10:21.018 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-09-16T04:10:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m1.133s 2024-09-16T04:10:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m2.919s 2024-09-16T04:10:21.018 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.117s 2024-09-16T04:10:21.101 INFO:teuthology.orchestra.run.smithi174.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T04:10:21.103 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-16T04:10:21.103 INFO:teuthology.orchestra.run.smithi174.stderr:real 0m1.248s 2024-09-16T04:10:21.103 INFO:teuthology.orchestra.run.smithi174.stderr:user 0m4.113s 2024-09-16T04:10:21.103 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m0.168s 2024-09-16T04:10:21.114 INFO:teuthology.orchestra.run.smithi016.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T04:10:21.328 INFO:teuthology.orchestra.run.smithi016.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T04:10:21.330 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-16T04:10:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m1.491s 2024-09-16T04:10:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m4.304s 2024-09-16T04:10:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m0.269s 2024-09-16T04:10:21.330 INFO:tasks.ceph:Archiving logs... 2024-09-16T04:10:21.331 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/remote/smithi016/log 2024-09-16T04:10:21.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:10:22.242 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/remote/smithi174/log 2024-09-16T04:10:22.243 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:10:22.895 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/remote/smithi194/log 2024-09-16T04:10:22.896 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:10:23.465 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T04:10:23.475 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T04:10:23.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:10:23.479 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:10:23.482 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:10:23.583 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T04:10:23.592 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T04:10:23.592 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:10:23.595 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:10:23.598 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:10:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:10:23.612 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:10:23.616 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:10:23.616 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-16T04:10:23.616 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv01.front.sepia.ceph.com 3 6 377 31 -332us[ -334us] +/- 40ms 2024-09-16T04:10:23.617 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 +1173us[+1171us] +/- 72ms 2024-09-16T04:10:23.617 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -621us[ -623us] +/- 81ms 2024-09-16T04:10:23.617 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:10:23.618 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:10:23.618 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:10:23.619 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-09-16T04:10:23.619 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 -323us[ -324us] +/- 40ms 2024-09-16T04:10:23.619 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 +1187us[+1187us] +/- 72ms 2024-09-16T04:10:23.619 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -620us[ -620us] +/- 81ms 2024-09-16T04:10:23.619 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:10:23.621 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:10:23.621 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-09-16T04:10:23.621 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 -317us[ -300us] +/- 40ms 2024-09-16T04:10:23.621 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 +1185us[+1185us] +/- 72ms 2024-09-16T04:10:23.621 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -614us[ -597us] +/- 81ms 2024-09-16T04:10:23.621 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:10:23.621 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T04:10:23.632 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T04:10:23.632 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T04:10:23.648 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T04:10:23.734 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T04:10:23.805 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T04:10:23.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:10:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:10:23.909 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:10:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:10:23.942 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:10:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:10:23.974 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T04:10:23.988 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T04:10:24.001 INFO:teuthology.task.internal:Duration was 590.323945 seconds 2024-09-16T04:10:24.002 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T04:10:24.013 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T04:10:24.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:10:24.015 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:10:24.018 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:10:24.057 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:10:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:10:24.059 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:10:24.556 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T04:10:24.556 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-09-16T04:10:24.557 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:10:24.583 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-09-16T04:10:24.583 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:10:24.624 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-09-16T04:10:24.624 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:10:24.653 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T04:10:24.654 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:10:24.657 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:10:24.667 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:10:24.722 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T04:10:24.723 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:10:24.729 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:10:24.737 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:10:24.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T04:10:24.882 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T04:10:24.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:10:24.909 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:10:24.934 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:10:24.960 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T04:10:24.970 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:10:24.972 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:10:24.977 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:10:24.994 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-09-16T04:10:24.998 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-09-16T04:10:25.023 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-09-16T04:10:25.045 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:10:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:10:25.077 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:10:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:10:25.093 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:10:25.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:10:25.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T04:10:25.118 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T04:10:25.119 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/remote/smithi016 2024-09-16T04:10:25.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:10:25.157 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/remote/smithi174 2024-09-16T04:10:25.157 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:10:25.201 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905966/remote/smithi194 2024-09-16T04:10:25.201 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:10:25.236 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T04:10:25.236 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:10:25.239 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:10:25.241 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:10:25.286 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T04:10:25.300 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T04:10:25.300 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T04:10:25.310 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T04:10:25.310 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:10:25.312 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:10:25.315 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:10:25.327 INFO:teuthology.orchestra.run.smithi016.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:10 /home/ubuntu/cephtest 2024-09-16T04:10:25.328 INFO:teuthology.orchestra.run.smithi174.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:10 /home/ubuntu/cephtest 2024-09-16T04:10:25.341 INFO:teuthology.orchestra.run.smithi194.stdout: 266054 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:10 /home/ubuntu/cephtest 2024-09-16T04:10:25.342 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T04:10:25.353 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T04:10:25.378 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} duration: 590.323944568634 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T04:10:25.378 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:10:25.465 INFO:teuthology.run:pass